Method and apparatus for image mosaicing

ABSTRACT

A method and apparatus is provided for capturing images using a camera or other imager having imaging sensitivity characteristics which vary across the imager&#39;s viewing angle. The imager&#39;s characteristics can be non-uniform with respect to exposure, color sensitivity, polarization sensitivity, focal distance, and/or any other aspect of image detection. The imager is rotated or translated in order to capture different portions of the scene being imaged. Because the imager is in multiple positions when the respective scene portions are captured, each scene portion is imaged by multiple portions of the imager&#39;s sensitivity profile.

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional PatentApplication entitled “Multidimensional Image Mosaics,” Serial No.60/220,025, which was filedi on Jul. 21, 2000 and is incorporated hereinby reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] This invention was partially made with U.S. Government supportfrom the National Science Foundation Research Award No. IIS-00-85864.Accordingly, the U.S. Government may have certain rights in thisinvention.

BACKGROUND OF THE INVENTION

[0003] 1. Field of the Invention

[0004] The present invention relates generally to image mosaicing, andmore particularly to methods and systems for using image mosaicing toenhance the dynamic range of images and/or to determine additionalcharacteristics of radiation signals received from scenes.

[0005] 2. Description of the Related Art

[0006] Major limitations of imaging systems (e.g., cameras) includelimited field of view, limited dynamic range, limited spectral (e.g.,color) resolution, and limited depth of field (i.e., limited range ofdistances at which scene points remain are adequately in focus in theimage plane). In addition, conventional imaging systems typicallymeasure only the intensity of incoming light as a function of thedirection from which the light is received, and are unable to measureother characteristics such as depth (e.g., the distance of objects fromthe camera), and the polarization state of the light—which would beuseful for remote recognition of materials, shapes and illuminationconditions, and for the analysis of reflections. Furthermore, thequality of the measurements made by conventional cameras tends to berelatively low. For example, in typical CCD cameras, the intensitydefinition has only 8 bits, and the spectral definition is very poor,consisting of only three broad band channels - typically red, green, andblue channels.

[0007] Even when attempts have been made to overcome the above-describedlimitations, the resulting system has been complex, and has addressedonly a narrow problem, while ignoring the other limitations. Forexample, imaging spectrometers provide high resolution in the spectraldimension, but do not extend the intensity dynamic range of the sensor.

[0008] A common way to obtain images having a large field of viewwithout compromising spatial resolution is by using “image mosaics.”Such technique involves combining smaller images, each of which covers adifferent view of the scene, to obtain a larger image having a widerfield of view. The method has been used in various scientific fieldssuch as radio astronomy, remote sensing by synthetic aperture radar(SAR), optical observational astronomy, and remote optical sensing ofthe Earth and other objects in the solar system. Recently, algorithmshave been developed to cope with arbitrary camera motions, and suchalgorithms have enabled image mosaics to be used in video cameras. Inregions where the smaller, component images overlap, the raw data can beprocessed to enhance its spatial resolution. However, conventional imagemosaic techniques are unable to enhance resolution (e.g., dynamic range)with regard to the spectrum, polarization, and brightness of each pixel.Depth is recoverable from image mosaics if parallax is introduced into asequence of images. However, parallax methods are usually less robustand more complex than methods which estimate depth using focus/defocuscues.

[0009] Nonlinear detectors have been used to extend the optical dynamicrange of images. For example, CMOS detectors have been manufacturedwhich: (1) yield an electronic output signal which is logarithmic withrespect to light intensity, or (2) combine two images having differentintegration times. The intensity dynamic ranges of such sensors tend tobe on the order of 1:106, which enables unsaturated detection of large(i.e., high irradiance) signals. However, the intensity information insuch a device is compressed, because in order to sample (sparsely) thehigh intensity range, the detector uses quantization levels which wouldotherwise be dedicated to the lower intensities. Thus, the output stillhas only 8-12 bits of intensity resolution.

[0010] Nonlinear transmittance hardware which has a lower transmittancefor higher light intensities can extend the dynamic range of any givendetector. However, the intensity dynamic range is still quantizedaccording to the limited definition of the detector—i.e., the 8 bits ofdefinition in an ordinary CCD are simply nonlinearly stretched to covera higher irradiance range. Consequently, the nonlinear compressionsacrifices resolution in the lower intensity range.

[0011] Automatic gain control (AGC) is common in video and digitalcameras, and is analogous to automatic exposures in still-image cameras.However, a major drawback of AGC is that its effect is global, and as aresult, the gain setting is likely to be too high for some portions ofthe-image, yet too low for other portions. For example, a bright pointis likely to be saturated if it is within a relatively dark image, and adim point is likely to be too dark for proper detection if it is withina relatively bright image. Image mosaics can be constructed fromsequences in which AGC adaptively changes the sensor gain as the sceneis scanned. However, although some enhancement of dynamic range has beenachieved by this technique, such methods still suffer from an inabilityto properly measure bright points in mostly dark images, and dark pointsin mostly bright images.

[0012] Mounting spatially varying optical filters on a camera is acommon practice in amateur and professional photography. However, suchfilters have primarily been used to alter raw images to produce specialvisual effects. Such filters have not been used in connection withresolution enhancement algorithms.

[0013] It has been proposed that the dynamic range of each pixel of animage can be enhanced by using a set of multiple, differently exposedimages. One such method involves estimating, for each pixel, the valuethat best agrees with the data from the multiple samples of the pixel.Another approach is to select, for each pixel, the value that maximizesthe local contrast. However, such approaches use a stationary camera tocapture the sequence of images, and consequently, provide no enlargementof the field of view.

[0014] An additional approach uses a mosaic array of small filters whichcover the detector array of the imager. Each filter covers a particulardetector pixel. The result is a spatially inhomogeneous mask whichmodulates the light impinging on the detector. In order to extend theintensity dynamic range, the sensor array can be covered with a spatialmosaic array of neutral (i.e., color independent) density filters.However, such a configuration sacrifices spatial resolution in order toextend the dynamic range. Spectral information can be obtained bycovering the detector with a mosaic array of color filters. However,such a configuration sacrifices spatial resolution in order to obtainsome spectral resolution (i.e., color information). In addition, adetector can be covered with a mosaic of linear polarizers oriented invarious different directions. However, such a configuration sacrificesspatial resolution for the polarization information.

[0015] High resolution spectral filtering has been obtained by coveringa detector array with a spatially varying spectral filter—i.e., a filterhaving a spectral passband which changes across the vertical and/orhorizontal viewing angle of the detector. In such a system, differentpoints in the field of view are filtered differently. The spectrum ateach point is obtained by scanning the camera's field of view across thescene. However, placing the filter directly on the detector arrayreduces the flexibility of the system by making it difficult to changethe effective characteristics of the spectral filtering or to measureother properties of the light received from the scene.

[0016] If the scene is scanned line by line with a linear scanner,spatial resolution is not sacrificed to obtain spectral information. Forexample, in trilinear scanners, each linear portion of the image issensed consecutively with red, green, and blue filters. Pushbroomcameras, which are often used in remote sensing work, operate similarly;each scene line is diffracted by a dispersive element onto a 2D detectorarray, and as a result, each line is simultaneously measured in multiplespectral channels. However such scanners and pushbrooms are limited toone-dimensional (1-D) scanning at constant speed. Furthermore, an imageformed by such a system is not foveated; the entire image is scannedusing the same detector characteristics. Accordingly, to capture asignificant field of view, numerous acquisitions need to be taken,because each acquisition captures only a 1-pixel wide column.

[0017] Images have been captured with different focus settings, and thencombined to generate an image with a large depth of field. An approachusing a tilted sensor has succeeded in capturing all scene points infocus while extending the field of view. However, this approach does notenhance the dynamic range of the image.

[0018] It is common practice in optices to revolve spatially varyingchoppers and reticles in front of, or within, an imaging system.However, such systems require the imager to have additional internal orexternal parts which move during image acquisition.

SUMMARY OF THE INVENTION

[0019] It is therefore an object of the present invention to provide animaging technique which provides both an enlarged field of view andenhanced brightness dynamic range.

[0020] It is a further object of the present invention to provide animaging technique which provides both an enlarged field of view andspectral, polarization, and/or depth information about a scene.

[0021] These and other objects are accomplished by the following aspectsof the present invention.

[0022] In accordance with one aspect of the present invention, a methodfor generating enhanced-resolution data comprises: A method for imaging,comprising: a first step of using an imager to perform a first set ofmeasurements for generating a first image value, the first set ofmeasurements including at least one measurement of an intensity of afirst radiation ray bundle from a first scene region, the firstradiation ray bundle having a first chief ray in a reference frame ofthe imager, the imager having a first intensity sensitivitycharacteristic with respect to radiation ray bundles having the firstchief ray, and the imager having a first dynamic range with respect tointensities of the radiation ray bundles having the first chief ray; asecond step of using the imager to perform a set of second measurementsfor generating a second image value, the second set of measurementscomprising at least one measurement of an intensity of a secondradiation ray bundle emanating from the first scene region, the secondradiation ray bundle having a second chief ray in the reference frame ofthe imager, the second chief ray being different from the first chiefray, the imager having a second intensity sensitivity characteristicwith respect to radiation ray bundles having the second chief ray, thesecond intensity sensitivity characteristic being different from thefirst intensity sensitivity characteristic, and the imager having asecond dynamic range with respect to intensities of the radiation raybundles having the second chief ray; and applying a mosaicing operationto the first and second image values, for generating a third image valuehaving associated therewith a third dynamic range of the imager withrespect to at least one of the intensities of the first and secondradiation ray bundles, the third dynamic range being greater than atleast one of the first and second dynamic ranges of the imager.

[0023] In accordance with an additional aspect of the present inventiona method for imaging, comprising: a first step of using an imager toperform a first set of measurements for generating a first image value,the first set of measurements comprising at least one measurement of anintensity of at least one selected polarization component of a firstradiation ray bundle emanating from a first scene region, the firstradiation ray bundle having a first chief ray in a reference frame ofthe imager, the imager having a first polarization sensitivitycharacteristic with respect to radiation ray bundles having the firstchief ray, and the first polarization sensitivity characteristiccomprising reduced sensitivity to signal components having polarizationangles outside a first angular range, the at least one selectedpolarization component of the first radiation ray bundle having apolarization angle within the first angular range; a second step ofusing the imager to perform a second set of measurements for generatinga second image value, the second set of measurements comprising at leastone measurement of an intensity of at least one selected polarizationcomponent of a second radiation ray bundle emanating from the firstscene region, the second radiation ray bundle having a second chief rayin the reference frame of the imager, the second chief ray beingdifferent from the first chief ray, the imager having a secondpolarization sensitivity characteristic with respect to radiation raybundles having the second chief ray, the second polarization sensitivitycharacteristic comprising reduced sensitivity to signal componentshaving polarization angles outside a second angular range, the at leastone selected polarization component of the second radiation ray bundlehaving a polarization angle within the second angular range, and thesecond angular range being different from the first angular range; athird step of moving the imager, comprising one of: rotating the imagerwith respect to the first scene region between the first and secondsteps; and translating the imager with respect to the first scene regionbetween the first and second steps; and using the first and second imagevalues to determine a polarization state of one of the first and secondradiation ray bundles.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Further objects, features, and advantages of the presentinvention will become apparent from the following detailed descriptiontaken in conjunction with the accompanying figures showing illustrativeembodiments of the present invention, in which:

[0025]FIG. 1 is a flow diagram illustrating an exemplary procedure forimage mosaicing in accordance with the present invention;

[0026]FIG. 2 is a flow diagram illustrating an exemplary calibrationprocedure in accordance with the present invention;

[0027]FIG. 3 is a flow diagram illustrating an additional exemplarycalibration procedure in accordance with the present invention;

[0028]FIG. 4 is a flow diagram illustrating an exemplary procedure forimage mosaicing in accordance with the present invention;

[0029]FIG. 5 is a block diagram illustrating an exemplary imagemosaicing technique in accordance with the present invention;

[0030]FIG. 6 is a block diagram illustrating an exemplary imagemosaicing system in accordance with the present invention;

[0031]FIG. 7 is a diagram illustrating an additional exemplary systemfor image mosaicing in accordance with the present invention;

[0032]FIG. 8 is a diagram illustrating yet another system for imagemosaicing in accordance with the present invention;

[0033]FIG. 9 is a diagram illustrating a further exemplary system forimage mosaicing in accordance with the present invention;

[0034]FIG. 10 is a diagram illustrating still another exemplary systemfor image mosaicing in accordance with the present invention;

[0035]FIG. 11 is a diagram illustrating an additional system for imagemosaicing in accordance with the present invention;

[0036]FIG. 12 is a diagram illustrating another exemplary system forimage mosaicing in accordance with the present invention;

[0037]FIG. 13 is a diagram illustrating still another exemplary systemfor image mosaicing in accordance with the present invention;

[0038]FIG. 14 is a diagram illustrating a further exemplary system forimage mosaicing in accordance with the present invention;

[0039]FIG. 15 is a diagram illustrating yet another exemplary system forimage mosaicing in accordance with the present invention;

[0040]FIG. 16 is a diagram illustrating a further exemplary system forimage mosaicing in accordance with the present invention;

[0041]FIG. 17 is a diagram illustrating an additional system for imagemosaicing in accordance with the present invention;

[0042]FIG. 18 is a diagram illustrating another exemplary system forimage mosaicing in accordance with the present invention;

[0043]FIG. 19 is a diagram illustrating a further exemplary system forimage mosaicing in accordance with the present invention;

[0044]FIG. 20A is a graph illustrating the spatial and intensity rangesof an exemplary procedure for image mosaicing in accordance with thepresent invention;

[0045]FIG. 20B is a graph illustrating the spatial and intensity rangesof an additional exemplary procedure for image mosaicing in accordancewith the present invention;

[0046]FIG. 21 is a graph illustrating the spatial and intensity rangesof yet another exemplary procedure for image mosaicing in accordancewith the present invention;

[0047]FIG. 22A is a graph illustrating a density profile of an exemplaryfilter and corresponding effective mask characteristic in accordancewith the present invention;

[0048]FIG. 22B is a graph illustrating a density profile of anadditional exemplary filter and corresponding effective maskcharacteristic in accordance with the present invention;

[0049]FIG. 22C is a graph illustrating a density profile of yet anotherexemplary filter and corresponding effective mask characteristic inaccordance with the present invention;

[0050]FIG. 23 is a diagram illustrating an exemplary system for imagemosaicing in accordance with the present invention;

[0051]FIG. 24A is a graph illustrating effective density profiles ofexemplary imager attachments in accordance with the present invention;

[0052]FIG. 24B is a graph illustrating logarithmic functions of theprofiles illustrated in FIG. 24A.

[0053]FIG. 25A is a graph illustrating a central wavelength profile ofan exemplary filter and corresponding effective mask characteristic inaccordance with the present invention;

[0054]FIG. 25B is a graph illustrating a central wavelength profile ofan additional exemplary filter and corresponding effective maskcharacteristic in accordance with the present invention;

[0055]FIG. 25C is a graph illustrating a central wavelength profile ofyet another exemplary filter and corresponding effective maskcharacteristic in accordance with the present invention;

[0056]FIG. 26A is a graph illustrating a cutoff wavelength profile of anexemplary filter and corresponding effective mask characteristic inaccordance with the present invention;

[0057]FIG. 261 is a graph illustrating a cutoff wavelength profile of anadditional exemplary filter and corresponding effective maskcharacteristic in accordance with the present invention;

[0058]FIG. 26C is a graph illustrating a cutoff wavelength profile ofyet another exemplary filter and corresponding effective maskcharacteristic in accordance with the present invention;

[0059]FIG. 27 is a graph illustrating a portion of a sensitivitycharacteristic of an exemplary imager having a high pass filter array inaccordance with the present invention;

[0060]FIG. 28 is a graph illustrating sensitivity characteristics of anexemplary imager having a narrow band filter in accordance with thepresent invention;

[0061]FIG. 29 is a graph illustrating sensitivity characteristics of anexemplary imager having a set of broad band filters in accordance withthe present invention;

[0062]FIG. 30A is a diagram illustrating an exemplary polarizing filterarray in accordance with the present invention;

[0063]FIG. 30B is a diagram illustrating the fabrication of an exemplarypolarizing filter in accordance with the present invention;

[0064]FIG. 30C is a diagram illustrating the fabrication of anadditional exemplary polarizing filter in accordance with the presentinvention;

[0065]FIG. 31 is a diagram illustrating the focal characteristics of anexemplary camera;

[0066]FIG. 32A is a graph illustrating a refraction profile of anexemplary refractive element in accordance with the present invention;

[0067]FIG. 32B is a graph illustrating a refraction profile of anadditional exemplary refractive element in accordance with the presentinvention;

[0068]FIG. 32C is a graph illustrating a refraction profile of yetanother exemplary refractive element in accordance with the presentinvention;

[0069]FIG. 33 is a diagram illustrating an exemplary arrangement of aset of optical elements in accordance with the present invention;

[0070]FIG. 34 is a diagram illustrating an exemplary optical element inaccordance with the present invention;

[0071]FIG. 35A is a diagram illustrating an exemplary procedure forimage portion registration in accordance with the present invention;

[0072]FIG. 35B is a diagram illustrating an additional exemplaryprocedure for image portion registration in accordance with the presentinvention;

[0073]FIG. 36A is a graph illustrating an attenuation profile of anexemplary mask in accordance with the present invention;

[0074]FIG. 36B is a graph illustrating a logarithmic characteristic ofthe attenuation profile illustrated in FIG. 36A;

[0075]FIG. 37 is a diagram illustrating a computer system for performingimage mosaicing algorithms in accordance with the present invention; and

[0076]FIG. 38 is a block diagram of a processor section for use in thecomputer system of FIG. 37.

[0077] Throughout the figures, unless otherwise stated, the samereference numerals and characters are used to denote like features,elements, components, or portions of the illustrated embodiments.Moreover, while the subject invention will now be described in detailwith reference to the figures, and in connection with the illustratedembodiments, changes and modifications can be made to the describedembodiments without departing from the true scope and spirit of thesubject invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE INVENTION

[0078] In accordance with the present invention, an imager—which caninclude an electronic still-image camera or a moving-image camera suchas a video camera—can be configured such that one or more of theimager's detection characteristics varies across the vertical and/orhorizontal viewing angle. For example, the left side of the field ofview of the imager can be configured to have characteristics whichdiffer from those of the right side of the field of view, or the topportion of the field of view can be configured to have characteristicswhich differ from those of the bottom portion of the field of view. Thenon-uniform (i.e., spatially varying) sensitivity characteristics of theimager can include, for example, sensitivity to scene brightness,sensitivity to light having a particular color, sensitivity to lighthaving a particular polarization angle, and/or focal distance (i.e., thedistance at which objects are in focus). By rotating an/or translatingthe imager between successive snapshots or frames, the successivesnapshots or frames can be combined to form a larger image having awider field of view. Such combining of different views of a scene can bereferred to as “image mosaicing.” In addition, if the motion of thecamera between snapshots or frames is sufficiently small, some regionsof the scene are captured multiple times, each time through a differentportion of the field of view of the camera. Because each portion of theimager's field of view has a different sensitivity characteristic, theresulting “multisampled” scene portions (i.e., portions sampled multipletimes) are captured using a variety of imaging sensitivitycharacteristics. Therefore, additional information can be obtained abouteach portion of the scene. For example, a scene portion can be sampledmultiple times using a wide range of intensity sensitivitycharacteristics—e.g., by capturing multiple frames while panningvertically or horizontally (or at any angle) across the scene with acamera having a spatially non-uniform attenuator mounted on the lens.Each portion of the scene is thus effectively captured with an enhanceddynamic range. In addition, spectral information about each sceneportion can be obtained by panning across a scene while taking multiplesnapshots using an imager having spatially non-uniform color sensitivitycharacteristics. Similarly, polarization information can be obtained byusing an imager having non-uniform polarization sensitivitycharacteristics (e.g., a camera with a non-uniform polarizing filter),and depth (i.e., distance) information can be obtained using an imagerhaving a non-uniform focal distance.

[0079] A system in accordance with the present invention can be viewedgenerally as having two parts, as illustrated in the block diagram ofFIG. 5. The system includes a hardware part which comprises an imager(e.g., a camera) 502 which is used to capture images 504. The images 504are processed by an image processing part executing software comprisingone or more image processing algorithms 506, which provide enhancedoutput images 508 and/or information 510 regarding the properties of thescene being imaged. FIG. 6 illustrates such a system in further detail.The imager 502 comprises one or more filters 604 which filter lightsignals (e.g., light bundles) 632 entering the imager 502. The imager502 includes imaging optics 606 and a detector array 608, such as a CCDor a CMOS image sensing array. The detector array 608 generates signalswhich can be processed by circuits 610. In order to rotate and/ortranslate the imager 502, the imager 502 can be mounted on a motorizedrotation and/or translation support 630 and/or can be carried by amoving platform such as an airplane.

[0080] Analog signals generated by the camera circuitry 610 areprocessed by an analog-to-digital converter 612 which converts theanalog signals into digital signals which can be stored in a framememory 614. The images are analyzed and processed using a processor 618which can be the processor of computer. The processor 618 executes thevarious algorithms in accordance with the present invention. Theprocessor 618 can also be used to “render” (i.e., generate) new imagesof the scene. A converter 616 can be used to convert images into aformat which can be processed by an external device 602 to generateoutput images 508. The external device 602 can include, for example, aconverter 620 for converting the images into a video signal, a printersignal, or other output format. The processor 618 can also extractinformation 622 regarding the properties of the scene being imaged.Examples of the aforementioned techniques are described in furtherdetail below.

[0081]FIG. 8 illustrates an example of an imager for performing imagemosaicing in accordance with the present invention. The imager 712includes a camera 702 having an aperture 710, an objective lens 802, andan image detector 708 which can be, for example, a CCD detector array.Also included is a spatially varying, neutral (i.e.,wavelength-independent) density filter 704 which, in the illustratedexample, has lower attenuation near its top portion 816, and greaterattenuation near its bottom portion 820. The imager 712 receivesradiation ray bundles 810, 812, and 814 (in this example, bundles oflight) from scene points A, B and C, respectively. In the referenceframe of the imager 712, there are a number of chief rays—e.g., rays804, 806 and 808—which define the respective directions from whichradiation ray bundles 810, 812 and 814 or other sets of radiationsignals are received by the imager 712. As is well known in optics, abundle of light is also typically considered to have a chief ray whichcan be used to define the path along which the bundle propagates. In theillustrated example, the chief rays of bundles 810, 812, and 814,respectively, are the chief rays 804, 806 and 808 of the imager 712.Furthermore, although only three chief rays 804, 806 and 808 areillustrated in FIG. 8, an imager will in theory have an infinite numberof chief rays in its reference frame. In addition, although theexemplary imager 712 illustrated in FIG. 8 is used primarily to imagelight, the techniques of the invention are applicable to the imaging ofany electromagnetic radiation or any other radiation, including, but notlimited to, infra-red (IR) radiation, X-ray radiation, syntheticaperture radar (SAR) signals, particle beams (e.g., electron beams forelectron microscopy), and acoustic (e.g., ultrasound) radiation.

[0082] The signal sets 810, 812, and 814 which respectively comprisesignals emanating from points A, B, and C of the scene are focused bythe objective lens 802 onto points A′, B′, and C′, respectively, on thedetector 708. The imager 712 can be rotated about a rotation vector ŷ tothereby enable the imager 712 to receive any one of the radiation raybundles 810, 812, and 814 along any one of the chief rays 804, 806, and808, or along any other chief ray in the field of view of the imager712. For example, a first snapshot can be taken with the imager 712oriented as illustrated in FIG. 8, in which case signal set 810 isreceived along chief ray 804 as illustrated. The imager 712 can then berotated in a counterclockwise direction such that ray bundle 810 isreceived along chief ray 806. An additional snapshot can then be taken.The imager 712 can then be further rotated such that signal set 810 isbeing received along chief ray 808, and a third snapshot can be taken.Accordingly, in the respective three snapshots, ray bundle 810 has beenreceived through three portions 816, 818 and 820 of the filter 704, andaccordingly, has been received at three points A′, B′, and C′ of thedetector 708. Furthermore, it is to be noted that although FIG. 8illustrates an imager 712 which is rotated in order to pan across ascene, the imager 712 can in addition, or alternatively, be translatedas illustrated in FIG. 7. In fact, any arbitrary motion can be used topan the imager 712 across a scene 706, thereby enabling the imager 712to receive radiation signals from a scene point A along any chief ray ofthe field of view of the imager 712. Once the multiple snapshots of thescene 706 have been taken, the snapshots can be used for an imagemosaicing procedure in accordance with the invention. For example, asdiscussed in further detail below with reference to FIG. 1, an imagemosaicing procedure can be used to enhance the dynamic range of imagesrepresenting the intensities of light signal sets received from variousregions of a scene 706.

[0083]FIG. 1 illustrates an exemplary procedure for image mosaicing inaccordance with the present invention. The procedure preferably includesa calibration step 100, examples of which are discussed in furtherdetail below. An imager (e.g., a camera with a spatially varyingattenuating filter)—item 712 in FIGS. 7-19, 23, and 31—is used toperform a first set or measurements of a scene being imaged (step 102).In the reference frame of the imager 712, there is a first chief ray(e.g., chief ray 804 illustrated in FIGS. 8-18), and for signalsreceived along the first chief ray—e.g., light bundles having the firstchief ray—the imager 712 has a first intensity sensitivitycharacteristic and a first dynamic range imaging system has a firstintensity sensitivity characteristic and a first dynamic range. Forexample, the first chief ray can correspond to a particular viewingdirection within the field of view of the imager 712, and this portionof the field of view can be covered by an attenuating filter having afirst amount of attenuation. The sensitivity of the imager with respectto a light bundle passing through a particular portion of theattenuating filter depends on the amount of attenuation of thatparticular portion of the filter. The dynamic range of the imager withrespect to the light bundle is typically determined by the dynamic rangeof the portion of the detector (e.g., the CCD cell) onto which thebundle is focused.

[0084] The output of the first measurement set is a first measurementvalue which can represent, for example, the intensity (i.e., thebrightness) of a first light ray bundle or other radiation emanatingfrom a first region or point within the scene. The first light raybundle has a chief ray which corresponds, or is identical, to the firstchief ray of the imager.

[0085] The imager 712 is moved (e.g., by rotation and/or translation) sothat a different portion of the scene is captured (step 104). The secondportion of the scene overlaps the first portion such that the firstscene point or scene region is still within the field of view of theimager 712, but a light ray bundle from the first scene region is nowbeing received along a second chief ray (e.g., chief ray 806 illustratedin FIGS. 8-18) in the reference frame of the imager 712. A secondmeasurement set is performed by the imager (step 106), therebygenerating a second measurement value. The second measurement valuerepresents the intensity of a second light ray bundle from the firstscene region. The imager 712 has a second intensity sensitivitycharacteristic with respect to radiation signal sets received along thesecond chief ray. For example, if the second chief ray passes throughattenuator having a second amount of attenuation which is different fromthe first amount of attenuation, the sensitivity of the imager 712 withrespect to the second light ray bundle is different from the sensitivityof the imager 712 with respect to the first light ray bundle.

[0086] A third measurement value is generated by applying a mathematicaloperation to the first and second measurement values (step 108). Forexample, if the imager 712 includes aCCD detector array 708 which isused to capture images, a first cell, or group of cells, of the CCDarray 708 are used to measure the first light ray bundle, therebygenerating the first measurement value, and a second cell, or group ofcells, of the CCD array 708 are used to measure the second light raybundle, thereby generating the second measurement value. Depending uponthe characteristics of the imager 712 with respect to signals receivedalong the first and second chief rays, the signal received by either thefirst or the second cell may be too bright or too dim, thereby causingthe signal to be outside the accurate range of the cell. If the firstmeasurement value indicates that the first light ray bundle has beenmeasured accurately, but the second measurement value indicates that thesecond light ray bundle has been measured less accurately, the secondmeasurement value can be discarded and the first measurement can be usedas the third measurement value which will serve as the output value forthe pixel representing the first scene region. If the characteristics ofthe region of the imager 712 impinged by the first chief ray aredifferent from those of the region impinged by the second chiefray—e.g., if light received along the first chief ray are more highlyattenuated than light received along the second chief ray - then thedynamic range of the imager 712 is effectively enhanced, because highintensity light will be measured accurately when received along thefirst chief ray, and low intensity light will be measured accuratelywhen received along the second chief ray. Because light emanating fromthe first scene region is capered along both chief rays, the light ismore likely to be measured accurately by at least one of the two regionsof the detector. Accordingly, the third measurement value can be viewedas having a third effective dynamic range which is greater than one orboth of the respective effective dynamic ranges of the first and secondmeasurement values.

[0087] In addition to the above-described steps for effectivelyenhancing the dynamic range of intensity measurements, the exemplaryimage mosaicing procedure illustrated in FIG. 1 can also include stepsfor measuring additional characteristics of light ray bundles receivedfrom the first scene region. Such additional characteristics caninclude, for example, spectral characteristics, polarization, and/orfocal characteristics—which can be used to infer distances from thecamera to various scene features. For example, a third measurement setcan be performed to measure the intensity of at least one spectralcomponent of a third light ray bundle from the first scene region (step110). The third measurement set generates a fourth measurement value.The third light ray bundle can be received along a third chief ray ofthe imager 712 which is configured to have a first spectral sensitivitycharacteristic with respect to light ray bundles having the third chiefray. The first spectral sensitivity characteristic preferably comprisesa bandpass characteristic having a first wavelength sensitivity band inorder to select the spectral component of the third light ray bundle. Inother words, the selected spectral component has a wavelength orwavelengths which are within the first wavelength sensitivity band. Theselected spectral component may or may not have sufficient energy togenerate a signal above the detector noise.

[0088] The camera is then rotated or translated (step 112), and a fourthmeasurement set is performed (step 114). The fourth measurement set,which generates a fifth measurement value, includes at least onemeasurement of the intensity of a spectral component of a fourth lightray bundle from the first scene region. The fourth light ray bundle isreceived along a fourth chief ray of the imager 712. The imager 712 hasa second spectral sensitivity characteristic with respect to radiationsignals having the fourth chief ray. The second spectral sensitivitycharacteristic preferably comprises a bandpass characteristic includinga second wavelength sensitivity band in order to select componentshaving wavelengths within the second wavelength sensitivity band.

[0089] In addition, an image mosaicing procedure in accordance with thepresent invention can be used to measure the polarization of lightemanating from the first scene region. In such a system, the thirdmeasurement set (step 110) comprises at least one measurement of anintensity of a selected polarization component of a third light raybundle. The third light ray bundle, which has the above-described thirdchief ray, comprises a third radiation signal from the first sceneregion. The imager 712 has a first polarization sensitivitycharacteristic with respect to radiation ray bundles received along thethird chief ray. The first polarization sensitivity characteristiccomprises a reduced sensitivity to light rays having polarization angleswhich are outside a first angular range. The selected polarizationcomponent of the third light ray bundle is detected because it has apolarization angle within the first angular range. The imager 712 isthen rotated or translated (step 112), and is then used to perform afourth measurement set, thereby generating a fifth measurement value(step 114). The fourth measurement set comprises at least onemeasurement of an intensity of a selected polarization component of afourth light ray bundle from the first scene region. The fourth lightray bundle is received along the fourth chief ray of the imager 712. Theimager 712 has a second polarization sensitivity characteristic withrespect to radiation ray bundles received along the fourth chief ray.The second polarization sensitivity characteristic comprises reducedsensitivity to signal components having polarization angles outside asecond angular range. The selected polarization component of the fourthlight ray bundle is detected by the imager 712 because the component hasa polarization angle within the second angular range.

[0090] Furthermore, it is possible to determine how far the first sceneregion is from the imager 712 by using an imager having non-uniformfocal characteristics. In particular, the imager 712 can be configuredto have a first focal characteristic with respect to light receivedalong a third chief ray, and a second focal characteristic with respectto light received along a fourth chief ray. A third measurement set canbe performed to generate a fourth measurement value, the thirdmeasurement set comprising at least one measurement of an intensity ofthe third light ray bundle from the first scene-region (step 110). Thethird radiation ray bundle is received along the third chief ray. Thefirst focal characteristic of the camera comprises a first focaldistance at which objects are in focus. The imager 712 is rotated ortranslated (step 112), and a fourth measurement set is performed togenerate a fifth measurement value (step 114). The fourth measurementset comprises at least one measurement of an intensity of a fourth lightray bundle from the first scene region. The fourth signal set isreceived along the fourth chief ray. The second focal characteristic ofthe camera comprises a second focal distance at which objects are infocus if imaged along the fourth chief ray. The second focal distance isdifferent from the first focal distance.

[0091] For image mosaicing using an imager having non-uniform intensitysensitivity characteristics, it can be beneficial to calibrate theimager by obtaining an estimate of how the intensity sensitivitycharacteristics of the camera vary across the field of view. Onetechnique for performing such a calibration is to capture a variety ofdifferent scenes and scene portions with the imager, and then add oraverage the measurements generated by each portion of the detector—e.g.,each detector cell. The result is a set of relative and/or scaled valuesrepresenting the imager characteristics along various chief rays. FIG. 2illustrates an example of such a calibration procedure 100 which can beused in the procedure illustrated in FIG. 1. Similarly to the othersteps of the procedure illustrated in FIG. 1, the sequence of steps 100illustrated in FIG. 2 is performed using an imager receiving first andsecond chief rays in the reference frame of the imager 712. The imager712 is used to perform measurements of intensities of a first pluralityof radiation ray bundles (e.g., light ray bundles) having the firstchief ray, to thereby generate a first set of calibration measurementvalues (step 202). The first set of calibration measurement values isused to determine a first estimate of the first intensity sensitivitycharacteristic of the imager 712 with respect to signals received alongthe first chief ray (step 204). The first estimate is determined bycalculating a sum and/or a mean of the first set of calibrationmeasurement values. The imager 712 is also used to perform measurementsof intensities of a second plurality of radiation ray bundles having thesecond chief ray, to thereby generate a second set of calibrationmeasurement values (step 206). The second set of calibration measurementvalues is used to determine a second estimate of the second intensitysensitivity characteristic of the imager 712 with respect to signalsreceived along the second chief ray (step 208). The second estimate isdetermined by calculating a sum and/or a mean of the second set ofcalibration measurement values.

[0092] In addition, the intensity sensitivity characteristic of theimager can be calibrated by tracking a scene region as it travels acrossthe field of view of the imager. For example, FIG. 3 illustrates acalibration procedure 300 which uses measurements of radiation raybundles (e.g., light ray bundles) received from a selected portion of ascene along multiple chief rays received by the imager. The calibrationprocedure 100 illustrated in FIG. 3 can be used in the image mosaicingprocedure illustrated in FIG. 3. In the illustrated calibrationprocedure 300, the imager 712 is used to perform a third measurementset, thereby generating a fourth measurement value (step 302). The thirdmeasurement set comprises at least one measurement of an intensity of athird light ray bundle from a second scene region. The third light raybundle is received along the first chief ray received by the imager. Theimager is then rotated or translated to enable the imager to receivelight rays from the second scene region along the second chief ray ofthe imager (step 304). A fourth measurement set is performed by theimager, thereby generating a fifth measurement value (step 306). Thefourth measurement set comprises at least one measurement of anintensity of a fourth light ray bundle from the second scene region. Thefourth light ray bundle is received along the second chief ray receivedby the imager. The fourth and fifth measurement values are used toestimate a mathematical relationship between the first intensitysensitivity characteristic of the imager 712 (i.e., the imager'ssensitivity with respect to light ray bundle having the first chief ray)and the second intensity sensitivity characteristic of the imager (i.e.,the imager's sensitivity with respect to light rays received along thesecond chief ray) (step 308). The mathematical relationship is estimatedby calculating either a difference of the fourth and fifth measurementvalues or a ratio of the fourth and fifth measurement values.

[0093] The above-described calibration procedure can be understood infurther detail as follows. Consider a neutral density mask M(x) whichchanges transmissivity only along the x axis. A scene point isrepresented in image k as image point x_(k); the linearized intensity ofimage point x_(k) is ĝ_(k). The same scene point is represented in imagep as image point x_(p); the intensity of image point x_(p) is ĝ_(p).Both points should obey the following relationship:

M(x _(k))ĝ _(p) −M(x _(p))ĝ _(k)=0.  (1)

[0094] Tracking some of the scene points in several images can providemany equations which the mask should obey at each image pixel x. Theequations can be expressed in matrix form as FM=0. An example of matrixF is: $\begin{matrix}{F = \begin{bmatrix}0 & 0 & g_{p = 1}^{x = 50} & 0 & \ldots & \ldots & \ldots & 0 & g_{p = 2}^{x = 3} & 0 & \ldots & \ldots & 0 & 0 & 0 \\0 & \ldots & 0 & \ldots & 0 & g_{p = 1}^{x = 87} & 0 & \ldots & \ldots & \ldots & 0 & g_{p = 2}^{x = 40} & 0 & \ldots & 0 \\\vdots & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \vdots \\g_{p = 15}^{x = 144} & 0 & \ldots & \ldots & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \ldots & 0 & g_{p = 18}^{x = 1} & 0\end{bmatrix}} & (2)\end{matrix}$

[0095] The frame number is indexed by p. Some points may not havereliable data, and therefore, it can be beneficial to impose additionalequations of smoothness in order to regularize the solution. Forexample, the equations for penalizing |∇²M|² can be formalized asseeking LM=0, where: . . . $\begin{matrix}{{L = \begin{bmatrix}1 & {- 2} & 1 & 0 & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & 0 \\0 & 1 & {- 2} & 1 & 0 & \ldots & \ldots & \ldots & \ldots & \ldots & 0 \\0 & 0 & 1 & {- 2} & 1 & 0 & \ldots & \ldots & \ldots & \ldots & 0 \\\vdots & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \quad & \vdots \\0 & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & 0 & 1 & {- 2} & 1\end{bmatrix}},} & (3)\end{matrix}$

[0096] Generally the above equations will contradict each other, andaccordingly, the algorithm seeks an optimal solution using a leastsquares method, a robust statistics method, or another method foroptimization. For the least squares method, the optimal solution is:$\begin{matrix}{{\hat{M} = {\arg \quad {\min\limits_{M}\left( {M^{\prime}A^{\prime}{AM}} \right)}}},} & (4) \\{where} & \quad \\{{A = \begin{bmatrix}F \\{\beta \quad L}\end{bmatrix}},} & (5)\end{matrix}$

[0097] and β is a parameter weighting the penalty for non-smoothsolutions relative to the penalty for disagreement with the data.

[0098] The nontrivial solution is found using singular valuedecomposition (SVD), and then max {circumflex over (M)} (i.e., themaximum of {circumflex over (M)}) is set equal to 1. The covariancematrix of M is estimated from the above equations:

Cov(M)=(A′A)⁻¹ {circumflex over (M)}′A′A{circumflex over (M)}(n _(r)*+1−l)⁻¹  (6)

[0099] where l is the number of elements of M, and n_(r) is the numberof rows in A. This can be viewed as a weighted least squares problem:rows that belong to L are weighted by β, while the rows that belong to Fare generally more accurate for more intense pixels (i.e., larger g).This is equivalent to using normalized rows, and then weighting each rowr by {square root}{square root over (Σ_(c)A²(r, c))}. Accordingly, thealgorithm uses n_(r)=Σ_(r,c)A²(r, c) thereby adding the squared weightsof each row.

[0100] The variance of M given by the diagonal of Cov(M) leads to theconfidence intervals of {circumflex over (M)}. Note that thisformulation is not in the log M domain. Thus, it does not penalizestrongly relative disagreements among the data at very low M, orfluctuations which may be relatively significant at low M. As a finalpost-processing step, smoothing of log {circumflex over (M)} is alsoperformed, which primarily affects the estimation of areas having stronglight attenuation.

[0101] It is to be noted that the self calibration of the mask can, infact, be based on the same image sequence which will ultimately beprocessed to obtain an mage representing the scene, in which case theestimate of the mask at a point x is not statistically independent ofthe measured signal at point x in a specific frame. However, it is to benoted that the estimation of M at each point is affected by hundreds, oreven tens of thousands of equations (rows of A), based on trackingnumerous points in several frames. Therefore, when registering andfusing the images, it is valid to assume that the signal at eachspecific sample is, for practical purposes, independent of the estimatedmask.

[0102] A self-calibration procedure in accordance with the presentinvention has been tested experimentally. The test used a commerciallyavailable linear variable density filter, 3 cm long, rigidly attachedapproximately 30 cm in front of the 25 mm lens of a linear CCD camera.The filter had a maximum density of 2 (corresponding to attenuation by afactor of 100), although the effective mask characteristics had a widerrange due to additional vignetting effects in the system. In the portionof the field of view in which the attenuation was lowest, M wasapproximately constant. The camera was rotated between frames so thateach point was imaged 14 times across the field of view. Using a roughestimate of M, the images were registered as the registration procedurediscussed below. Then, more than 50,000 equations were generated basedon random corresponding unsaturated and non-dark points. The equationswere used to determine the characteristic of the mask with a resolutionof 614 pixels. The estimated mask function generated by theabove-described self calibration procedure is illustrated in FIG. 36A.The logarithm, of the function is illustrated in FIG. 36B.

[0103] M can also be determined as a solution for a best fit, to thedata, of a parametric curve such as a polynomial, sigmoid, or splinecurve. Furthermore, alternatively, or in addition, to using the MSEcriterion, other methods of optimization, such as robust statistics anditerative projection methods, can be used. Because M is multiplicative,log M can be used as an additive parameter which can be optimized usinga linear optimization formalism in conjunction with the estimation of Iat each scene point.

[0104] In addition, if the scene I changes only along the x axis (a 1-Dsignal), and each “frame” p is translated by an amount tp relative tothe global coordinate system, then the sum of square errors is:$\begin{matrix}{\sum\limits_{p}{\sum\limits_{x}\left( {{{\hat{g}}_{p}(x)} - {{M(x)}{I\left( {x + t_{p}} \right)}}} \right)^{2}}} & (40)\end{matrix}$

[0105] for ĝ_(p)(x) not saturated.

[0106] The algorithm optimizes the above error function with respect tothe translation parameters t_(p), the mask M, and the variables LAdditional parameters which can be adjusted as part of the optimizationprocess can include: (1) the smoothness of M (or the number ofcoefficients needed to parameterize M), and (2) the smoothness of I, forthe particular amount of motion between frames (i.e., how much t_(p)differs from t_(p+1)). I and M are preferably constrained to bepositive. The sum of the square errors in Eq. (7) can be weighted, e.g.,as a function of M (making the problem non-linear), or the error cancalculated in the log domain to make the above equation a linearoptimization problem for unknown I and M.

[0107] In addition the mask can be calibrated iteratively. For example,the calibration procedure can begin with an initial estimated value foreach point of the mask, and the initially estimated mask function can beused to estimate a fused value of the intensity of each image pixel. Thefused image values are used to calculate an improved estimate for themask using the above-described calibration procedure. Then, the improvedmask estimate is used to derive a better estimate for the fused image,and so on.

[0108] In accordance with the present invention, an imager havingspatially varying polarization sensitivity characteristics can be usedto obtain information regarding the polarization of radiation (e.g.,light or other electromagnetic radiation) emanating from a scene,regardless of whether the imager also has a spatially varying intensitysensitivity characteristic. An example of such a procedure isillustrated in FIG. 4. The illustrated procedure 400 uses an imager 712having a first polarization sensitivity characteristic with respect toradiation signal sets received along a first chief ray of the imager712, and a second polarization sensitivity characteristic with respectto radiation received along a second chief ray of the imager 712. Forexample, the imager 712 can comprise a camera 702 with a non-uniformpolarizing filter which admits light having a first polarization anglein a first portion of the field of view, and which admits light having asecond polarization angle in a second portion of the field of view. Anexample of such a filter is illustrated in FIG. 30A. The illustratedexemplary filter 3010 comprises three polarizing portions 3002, 3004,and 3006 which transmit light having polarization angles 3020, 3022, and3024, respectively. The filter 3010 also includes a portion 3008 whichpasses light of any polarization through it. The filter 3010 illustratedin FIG. 30A can be mounted externally or internally to a camera 702 toform an imager 712 having a polarization sensitivity characteristicwhich varies between the left and right sides of the field of view. Whenthe imager is panned across a scene, and multiple snapshots or frames ofthe scene are captured, each scene point is captured through more thanone of the portions 3002, 3004, 3006, and 3008 of the spatially varyingpolarizing filter 3010.

[0109] A spatially varying polarizing filter can be fabricated in anumber of different ways. For example, in FIG. 30B, such polarizingfilter 3012 is cut from a disk 3018 of material in which thepolarization angle of transmitted light is in a direction azimuthal tothe disk 3018. In FIG. 30C, a spatially varying polarizing filter 3016is cut from a disk 3014 of material which transmits light having apolarization angle which is in a direction radial to the disk.

[0110] An imager 712 which includes one of the above-described filterswill typically have spatially varying polarization sensitivitycharacteristics. In any case, regardless of how such an imager isformed, in the procedure 400 illustrated in FIG. 4, the imager 712 isused to perform a first measurement set, thereby generating a firstmeasurement value (step 402). The first measurement set comprises atleast one measurement of an intensity of at least one selectedpolarization component of a first light ray bundle from a first regionof the scene. The first light ray bundle has a chief ray correspondingto a first chief ray received by the imager. The first polarizationsensitivity characteristic of the imager—with respect to a light raybundle having the first chief ray—comprises reduced sensitivity to lightcomponents having polarization angles outside a first angular range. Theselected polarization component of the first signal is detected by theimager, because it has a polarization angle within the first angularrange. The imager is rotated or translated in order to capture adifferent portion of the scene (step 404). The imager is used to performa second measurement set, thereby generating a second measurement value(step 406). The second measurement set comprises at least onemeasurement of an intensity of at least one selected polarizationcomponent of a second light ray bundle from the first scene region. Thesecond region has a chief ray which corresponds to the second chief rayreceived by the imager. The second polarization sensitivitycharacteristic of the imager—with respect to the light ray bundle havingthe second chief ray—comprises reduced sensitivity to signal componentshaving polarization angles outside a second angular range. The selectedpolarization component of the second light ray bundle is detectedbecause it has a polarization angle within the second angular range.

[0111] Once a set of snapshots and/or video frames have been captured,the resulting image data can be analyzed in a number of different ways.It is desirable to measure the properties of the scene with as muchprecision and accuracy as possible, but it is also desirable to performthe necessary measurements and computation in the most efficient mannerpossible. There is a tradeoff between the quantity of data acquired andthe time and computing power required to capture and analyze the data.Greater precision can be achieved by taking a greater number ofsnapshots or frames per unit change in viewing angle. However, thecapture and analysis of the resulting greater quantity of data is moretime consuming and expensive. There are typically diminishing returnsfrom capturing a larger number of images, because in practice, the sceneproperties being measured generally have limited degrees of freedom, andconsequently, excessive sampling of the scene can result in redundantdata. Furthermore, the effects of an externally mounted filter aretypically somewhat blurred because the filter is defocused, andaccordingly, using an extremely small angular change between snapshotsmay not result in enough additional information to justify theadditional time and expense required to capture and process the data.Therefore, image mosaicing procedures in accordance with the presentinvention should preferably balance the tradeoff between additionalinformation and additional time and expense. Methods for determiningefficient and practical frame rates per unit change in viewing angle arediscussed below with respect to image mosaicing procedures utilizingimagers having spatially varying sensitivity characteristics withrespect to intensity, wavelength, polarization, and depth

[0112] For example, consider an exemplary imager having a spatiallyvarying intensity sensitivity characteristic resulting from a maskmounted on a camera. Let the transmissivity of the mask be M, and letthe light intensity at a detector without the mask be I. Then the lightfalling on the detector after filtering is:

g(x,y)=M(x,y)I(x,y).  (8)

[0113] When viewing a scene having high contrast, it is typically usefulto define the intensity in terms of orders of magnitude or octaves. Forthis reason, camera aperture “f-stops” are typically arranged in octavessuch that each “stop” increase corresponds to a doubling of the measuredintensity. In digital camera sensors this corresponds to a shifting of 1bit in the binary representation of the measurement. For example, if an8-bit camera measures the light intensity at a pixel position in animage as 00011010, then an increase of one stop will result in acorresponding reading of 0011010(0) in a second image, where the newleast significant bit is the information added by the new image.

[0114] Consider, for example, the optimal mask for achieving an evendivision of orders of magnitude is one in which the attenuation octaveschange linearly across the field of view—i.e., the attenuation changesexponentially. Then, log₂M(x) is proportional to x. In thisconfiguration, a constant scanning increment (e.g., a sequence of equalchanges in the irnager's viewing direction) will yield a constant changein the order of magnitude of the measured intensity, and all intensityranges will be sampled equally. Such a behavior can be approximatelyachieved by attaching a linear variable density filter to the camera atsome distance in front of the lens. It should, however, be noted thatdue to vignetting, perspective, and lens distortions, the linearity ofthe filter density will not be exactly conserved in log M(x).

[0115] Let I be the light intensity that falls on the detector (i.e.,the irradiance) when the transmittance of the filter is at its maximum(i.e., M=1). A specific linear variable filter used in conjunction witha specific detector determines the minimal and maximal bounds of thescene radiance that can be sensed by the system without saturation. Letthe minimal irradiance that can be sensed by the detector (for the givencamera specifications) above the detector's noise in darkness beI_(min)^(detector).

[0116] This determines the minimum irradiance that can be sensed by theentire system. Let the maximum irradiance that the detector can measurewithout saturation be I_(max)^(detector).

[0117] The optical dynamic range of the detector in terms of octaves isthen: $\begin{matrix}{{DR}^{detector} = {\log_{2}\frac{I_{\max}^{detector}}{I_{\min}^{detector}}}} & (9)\end{matrix}$

[0118] Typically, DR^(detector)=8 bits. The maximum irradiance that theentire system can sense without being saturated is when the detectoryields its maximum output under the strongest attenuation—i.e., with thesmallest value of the mask M:I_(max)^(system) = I_(max)^(detector)/min   M.

[0119] Therefore, the total optical dynamic range of the system is:$\begin{matrix}\begin{matrix}{{DR}^{system} = {{\log_{2}\frac{I_{\max}^{system}}{I_{\max}^{detector}}} = {{DR}^{detector} - {\log_{2}\left( {\min \quad M} \right)}}}} \\{= {{DR}^{detector} + {{\log_{2}\left\lbrack {\max \left( {1/M} \right)} \right\rbrack}.}}}\end{matrix} & (10)\end{matrix}$

[0120] At issue is how to perform the most efficient sampling of sceneinformation, assuming that the captured images will have an irradiancerange between I_(min)^(detector)  and  I_(max)^(system).

[0121] Changing the viewing direction is effectively equivalent tochanging the aperture stop through which a point is seen (in terms ofthe light power allowed into the detector), and each change of a fullstop (i.e. a factor of 2 change of attenuation) is equivalent to a 1 bitshift in the binary representation of the measurement within the dynamicrange DR^(detector) of the detector. For some applications, it may bepreferable to require that minimal information is lost in the scan. Insuch cases, when the scan is complete, no bits should be “missed”between those acquired at each frame increment. However, forapplications requiring less redundancy, it is probably sufficient to usea “lossy” scan in which it is only required that when the scan iscomplete, no point is saturated and all points are above the detector'sthreshold (as long as (as  long  as  I ≥ I_(min)^(detector)).

[0122] Such a procedure yields the most efficient scan, in which theoptical dynamic range of the measurements is extended maximally at eachincrement.

[0123] For example, consider an imaging system in whichlog₂[max(1/M)]<DR^(detector), and in which and DR^(detector)=8 bits andmin M=⅛, hence DR^(system)=11 bits. Let a scene point yield the binaryvalue 11111111 when M=1 and 10111001 when M=½. Then the formermeasurement is saturated while the latter is not. Accordingly, for M=⅛(a shift of 2 more bits to the right) the measurement will yield00101110. The 11 bits representation of the irradiance is thus0010111001(0). Assuming that the scene images may contain irradiancevalues throughout the optical dynamic range of the system, the leastredundant scan which ensures minimal loss of information (bits) is ascan in which each point is sampled four times, at one-octaveincrements. If a linear variable filter is used, then there is a linearrelationship between: (1) the displacement between consecutive frames,and (2) the attenuation with which a particular scene region is imaged.

[0124] For applications in which less dynamic range is required, it may,for example, be sufficient to perform, for each point, a singlemeasurement which is unsaturated, yet which is above the detector'sthreshold if I ≥ I_(min)^(detector).

[0125] In such a case, it may be sufficient to use only M=1 and M=⅛.This yields the value 00101110(000) for the above-described, exemplarypixel, while yielding a value such as, for example, 00000000011 for arather dark point in the scene. In such a system, any point can bemeasured using one extreme value of the mask, and then using the otherextreme value, while skipping the intermediate mask values. In otherwords, it is sufficient to sample each point only twice. The inter-frameincrement can be large, provided that it is less than half the length ofthe frame, and provided that the transition region between M=1 and M=⅛is less than the inter-frame increment.

[0126] Now, consider a system in which log₂[max(1/M)]≧DR^(detector).Including the mask, the system's optical dynamic range is at leastdouble that of the detector alone. Sampling each point at one-octaveincrements ensures a minimal loss of bits. For a lossy, but mostefficient, scan it is preferable to measure each point withoutoverlapping intensity information between consecutive scan increments.For example, if DR^(detector)=8 bits and DR^(system)=16 bits then twosamples of each point suffice to acquire the entire range of values. Thefirst sample senses, in high definition, each point for which thequantizer of the system produces an image value—as scaled byI_(min)^(detector)−

[0127] which is smaller than 256, and the second sample captures,without saturation, each point for which the image value is larger than255 but smaller than 65536.

[0128] Generally, for a “lossless” efficient scan, each point should besampled 1+log₂[max(1/M)] times. For the most efficient scan, each pointshould be sampled ξ=ΠDR^(system)/DR^(detector)┐ times where ξ is alwaysrounded up to the nearest integer. In practice it may be desirable touse the foregoing figures merely as a rule of thumb, and use a somewhatdenser sampling rate, because the redundancy associated with densersampling enables less noisy irradiance estimation, and furthermoreassists in stable registration of the images within an image sequence.Moreover, it is to be noted that the above-described scene samplingprocedures are not the only ways to sample scenes; other periodic and/ornon-periodic sampling intervals can also be used.

[0129] Image mosaicing procedures in accordance with the invention canalso be used to achieve “super-resolution” in the intensity domain,because a smoothly varying mask acts as an analog computational device,and can therefore, in many cases, provide better quantization accuracythan a single, quantized image measurement. For example, consider animager having an integer quantizer for which the image values 96.8 and97.2 are indistinguishable—i.e., both image values will yield an outputvalue of 97. By attenuating the signal by a factor of 2—i.e., by usingM=½—the imager outputs the values 48 and 49, respectively, thusproviding enhanced discrimination. In another example, 95.8 and 96.2 canbe distinguished using the results of sensing with M=2⁻⁶, that is, using7 images. In general, if more images are used, the intensity resolutionof the fused estimate is better. [Specifically, how are the multiplesamples combined to give an accurate result here?] This is anotheradvantage of sampling the scene more densely than the rates discussedabove. [Which rates?]

[0130] For image mosaicing used to measure the spectral characteristicsof a scene, the optimal spacing of selected viewing angles depends uponhow quickly the intensity vs. wavelength function of the detector eachpixel position varies with respect to wavelength. This consideration issimilar to sampling issues in the context of many types of signals: ifthe samples are too sparse to sense the signal fluctuations, aliasingoccurs. Hence, if the spectrum of each scene point or scene region issmooth—as is the case for blackbody radiation, it may be sampledsparsely. On the other hand, if the spectrum contains fine lines such asemission and absorption lines of atomic vapors, the spectrum should besampled densely, and accordingly, the amount of rotation or translationof the imager between frames should be small.

[0131] However, even if the spectrum of light coming from the scene hasnarrow spectral bands, when these bands are measured, the measuredprofile is likely to be smoothed by the defocus blurring of anexternally mounted filter. Consider, for example, a linearly varyingspectral filter. The blurring of the measurements can be modeled as aconvolution of the spectrum s(λ) with a window function h(λ) of widthΔλ_(blur). Let S(ν_(λ)) and H(ν_(λ)) be the Fourier transforms of s(λ)and h(λ), respectively. His a lowpass filter having a cutoff frequencywhich is ≈1/Δλ_(blur). If the filter's transmission characteristic Hextends infinitely in frequency, the cutoff frequency can be defined as:(1) the frequency at which the signal energy is very low, (2) the firstzero of H, or (3) a quantity proportional to the standard deviation ofthe function H. In any case, this “cutoff” frequency will be≈1/Δλ_(blur). The required sampling interval is:

λ(x)_(sample)≈Δλ_(blur)/2.  (11)

[0132] The above sampling criterion can be expressed in terms of thephysical dimensions of the system. For a linear variable filter oflength L, $\begin{matrix}{{{\lambda (x)} = {\frac{\lambda_{\max} + \lambda_{\min}}{2} + {\frac{B}{L}x}}},\quad {x \in \left\lbrack {{- \frac{L}{2}},\frac{L}{2}} \right\rbrack},} & (12)\end{matrix}$

[0133] where λ_(max) and λ_(min) are the maximum and minimum wavelengthsthat the entire filter passes, respectively, and B≡λ_(max)−λ_(min) in isthe total bandwidth of the filter. If the model for the filter defocusblur is convolution by a window kernel of width Δx_(blur), thenΔx_(blur)=BΔx_(blur)/L. Hence: $\begin{matrix}{{\Delta \quad \lambda_{sample}} \approx {\frac{B\quad \lambda \quad x_{blur}}{2L}.}} & (13)\end{matrix}$

[0134] For example, consider the imager 712 illustrated in FIG. 8. Ifthe imager 712 is focused at infinity, then the external filter 704 isblurred by kernel having a width that is equal to the width of theobject light beam (e.g., beam 810, 812, or 814) passing through theaperture 710. Therefore, for the illustrated arrangement of lens 802 andexternal filter 704, Δx_(blur)=D, where D is the diameter of theaperture 710. Therefore: $\begin{matrix}{{\Delta \quad \lambda_{sample}} \approx {\frac{BD}{2L}.}} & (14)\end{matrix}$

[0135] The above result can be explained qualitatively as follows. Ifthe band B is stretched over a long filter so that the blur-kernel isinsignificant in size—i.e., D<<L—then the fine details of the originalspectrum (which may have sharp lines) remain, and thus the samplingshould preferably be dense. Quantitatively, it can be seen from that Eq.(14) that the sampling period Δλ_(sample) becomes small when D<<L.

[0136] Suppose that the imager 712 rotates with an angle increment ofΔφ_(sample) between image acquisitions. Let the filter 704 be situatedat a distance A from the rotation axis—which can be, for example, thecenter of projection O in the system illustrated in FIG. 8. Furthermore,let the filter 704 be normal to the optical axis. Each ray of lightpropagating from the scene to the center of projection is then displacedby Δx_(sample)≈AΔφ_(sample) sample between samples, assuming that thetotal angle θ of the field of view is small—i.e., assuming that sin θ≈θ.Referring to Eq. (11), $\begin{matrix}{{\Delta \quad \lambda_{sample}} \approx {\frac{B}{L}\Delta \quad x_{sample}} \approx {\frac{B}{L}A\quad \Delta \quad {\varphi_{sample}.}}} & (15)\end{matrix}$

[0137] Comparing Eqs. (14) and (15), and noting that the f-stop numberf#≡F/D where F is the focal length, it can be seen that: $\begin{matrix}{{{{\Delta \quad \varphi_{sample}} \approx \frac{D}{2A}} = \frac{F}{2{Af}\#}},} & (16)\end{matrix}$

[0138] irrespective of the dimensions or total bandwidth of the filter.A qualitative explanation corresponding to Eq. (16) is that a largeraperture D introduces larger defocus blur; accordingly, fewer samplesare needed, and therefore, Δφ_(sample) can be increased. Furthermore, asmaller distance A between the filter and the center of projectionresults in a smaller change in the wavelength λ of a light ray passingthrough the center of projection O for a given rotation increment,thereby decreasing the resulting effective wavelength sampling period.Therefore, for a desired Δλ_(sample), Δφ_(sample) should preferablyincrease as A decreases.

[0139] Eq. (16) demonstrates an advantage of using an externally mountedfilter, rather than placing a filter directly on the detector array:with an externally mounted filter, the user can select his/her desiredframe sampling rate simply by changing either the lens aperture size orthe distance of the external filter from the lens. If the applicationrequires high resolution with respect to spectral content, then thefilter is preferably placed farther away from the center of projection,and the aperture size is preferably small. On the other hand, if theuser would like to scan the scene quickly—which generally entails fewerframes for a given field of view—then Δφ_(sample) should be large.Accordingly, the aperture should be large and/or the filter should bepositioned relatively close to the lens.

[0140] As an additional example, consider an application which requiresa 360° panoramic view of a scene, and assume that the imager being usedincludes a filter of length L which covers the entire field of view ofthe imager. The length of the imager is denoted by L_(d). Then,F/A=L_(d)/L. Therefore: $\begin{matrix}{{\Delta \quad \varphi_{sample}} \approx {\frac{1}{2f\#}{\frac{L_{d}}{L}.}}} & (17)\end{matrix}$

[0141] In order to capture a 360° panorama, the required number ofimages is: $\begin{matrix}{N_{360{^\circ}\quad {panoram}} = {\frac{2\quad \pi}{\Delta \quad \varphi_{sample}} \approx {\frac{4\quad \pi \quad f\# L}{L_{d}}.}}} & (18)\end{matrix}$

[0142] It is to be noted that although the foregoing analysis is basedupon assumptions of a simple lens camera system, small angles, anexternal bandpass filter mask which is linear as a function ofwavelength, the general principles applied above can also be used toderive similar relations for other systems. Furthermore, for an imagerhaving a simple lens, or for any other system, the sampling intervalscan be different, and can in fact be non-periodic.

[0143] In addition, a set of images captured by an imager having aspatially varying polarization sensitivity characteristic can beanalyzed to determine the polarization characteristics of radiation(e.g., light) emanating from a scene. The polarization state of thelight coming from a scene point has four degrees of freedom—i.e., the 4Stokes parameters, which are well known in the art. Thus, if a detectormakes four or more measurements of light emanating from a scene point,each measurement having different polarization filtering, it is possibleto estimate the polarization state of the light. In many cases, ellipticand circular polarization states can be neglected, leaving only threedegrees of freedom: intensity, partial polarization, and polarizationplane (i.e., polarization angle). Let a linear polarizer be tuned topass the polarization component at angle α about the optical axis. Letthe polarization plane of the incident light be at angle θ. Let theintensity of the incident light be I and its partial polarization be P.Then, the intensity of light passing through the filter is:

g=C+A cos[2(α−θ)],  (19)

[0144] where C=½ and A PC. In this case three measurements generallysuffice to estimate the polarization state of the light. For example, ifin each measurement (corresponding to frame p) only the polarizer angleα_(p) changes, then $\begin{matrix}{{{\left\lbrack {1\quad {\cos \left( {2\alpha_{p}} \right)}\quad {\sin \left( {2\quad \alpha_{p}} \right)}} \right\rbrack \begin{bmatrix}C \\A_{c} \\A_{s}\end{bmatrix}} = g_{p}},} & (20)\end{matrix}$

[0145] where A_(c)=A cos(2θ) and A_(s)=A sin(2θ). Thus: $\begin{matrix}{{{M\begin{bmatrix}C \\A_{c} \\A_{s}\end{bmatrix}} = \overset{\rightarrow}{g}},\quad {where}} & (21) \\{{M = \begin{bmatrix}1 & {\cos \left( {2\alpha_{1}} \right)} & {\sin \left( {2\alpha_{1}} \right)} \\1 & {\cos \left( {2\alpha_{2}} \right)} & {\sin \left( {2\alpha_{2}} \right)} \\\vdots & \vdots & \vdots \\1 & {\cos \left( {2\alpha_{m}} \right)} & {\sin \left( {2\alpha_{m}} \right)}\end{bmatrix}},} & (22)\end{matrix}$

[0146] and m is the number of frames in which the scene point wasmeasured, and $\begin{matrix}{\overset{\rightarrow}{g} = {\begin{bmatrix}g_{1} \\g_{2} \\\vdots \\g_{m}\end{bmatrix}.}} & (23)\end{matrix}$

[0147] Neglecting the elliptic polarization component, Eq. (21) can beinverted using m=3 equations: $\begin{matrix}{\begin{bmatrix}C \\A_{c} \\A_{g}\end{bmatrix} = {M^{- 1}{\overset{\rightarrow}{g}.}}} & (24)\end{matrix}$

[0148] Eq. (24) can be used to calculate the values of the lightintensity I, the partial polarization P, and the angle of the plane ofpolarization θ. If, for example, the imager employs a filter coveringthe field of view φ_(FOV), and the polarization filtering changes acrossthe field of view φ_(FOV), then between each frame, the imager shouldpreferably change its viewing direction by Δφ_(sample)≈φ_(FOV)/3.

[0149] It is to be noted that in some cases, two measurements of thepolarized light from each scene point suffice to gain polarizationinformation regarding the scene. This is particularly the case if thereis some a priori information available regarding the polarization. Sucha priori information can include information regarding the polarizationproperties of typical scene objects which can be expected to be presentin the scene.

[0150] The polarization angle of detected radiation need not be the onlysensitivity characteristic which changes between frames. For example,the total attenuation of the radiation (which affects I) can alsochange. Moreover, a polarizer used in an imager need not be completelypolarizing, but can merely have a bias in favor of one polarizationcomponent over another. In such cases, the analysis is similar, but eachrow of the matrix M must be normalized to compensate for the attenuationassociated with the row. In some conventional systems, polarization ismeasured by using polarizing beam-splitters to split a light beam andsend the resulting separate components to separate sensors. Othersystems employ a rotating linear polarizer, or an electricallycontrolled polarizing apparatus such as a liquid crystal element.However, in accordance with the present invention, polarizationmeasurements can be performed using an imager having a spatially varyingpolarization sensitivity characteristic—e.g., a camera with a spatiallyvarying polarizing filter. Such an imager allows polarizationmeasurements to be performed without adding additional sensors, withoutmoving any of the internal parts of the imager during the imageacquisition, and without using an electronic filter. The avoidance ofadditional moving, internal parts reduces energy consumption andincreases reliability.

[0151] In performing an image mosaicing procedure in accordance with theinvention, it is particularly advantageous to employ an imager 712comprising a spatially non-uniform filter 704 mounted externally on acamera 702, as illustrated in FIGS. 7 and 8, because such an arrangementallows the imager 712 to be easily modified by merely modifying orreplacing the externally mounted filter 704. Additional examples ofimagers having externally mounted filters are illustrated in FIGS. 9 and10. The imager 712 illustrated in FIG. 9 includes an externally mountedspatially varying filter 902 Ai is curved such that every portion of thefilter 902 has approximately the same distance from the center ofprojection O of the aperture 710. In the system illustrated in FIG. 9,the projections, onto the aperture 710, of any two equal-sized portionsof the filter 902—e.g., portions 904 and 906—have approximately the samesize. Therefore, the sensitivity characteristics of the imager 712depend primarily upon the characteristics and/or thickness of thematerial from which the filter 902 is formed.

[0152] The system of FIG. 10 includes a filter 1002 which is orientedsuch that ihe distance between the filter 1002 and the center ofprojection O of the aperture 710 varies with viewing angle. Inparticular, a first portion 1004 of the filter 1002 is closer to thecenter of projection O than a second portion 1006 of the filter 1002. Asillustrated in FIG. 10, portion 1006 is oriented at an angle 1010 withrespect to chief ray 808 which passes through both portion 1006 and thecenter of projection O. Portion 1004 is oriented at a different angle1008 with respect to chief ray 804 which passes through both portion1004 and the center of projection O. Angle 1010 is smaller than angle1008, and as a result, although portion 1006 is larger in area thanportion 1004, the two portions 1004 and 1006 have the same area ofprojection upon the center of projection O of the aperture 710.

[0153] In the case of an attenuating filter, the effective attenuationof a particular portion of the filter increases with the sharpness ofthe angle at which light passes through the filter. In the case of awavelength-selecting filter such as an interference filter, thewavelength that is transmitted by the filter can depend upon the angleof incidence. Therefore, an externally mounted filter can beparticularly advantageous because, if the filter is either flexible ormovably mounted, the angular orientation of the filter can be easilymodified, thereby enabling easy modification of the characteristics ofthe imager.

[0154] Although the foregoing discussion has emphasized the use ofcameras having external filters, a spatially varying filter need not beexternal to the camera. For example, FIG. 11 illustrates an image 712having a filter 1102 which is positioned directly on the detector 708.An advantage of placing the filter 1102 directly on the detector 708 isthat because the detector 708 is at the focal plane of the lens 802, andbecause the filter 1102 is extremely close to the detector 708, there isvery little blurring of the characteristics of the filter 1102. As aresult, the characteristics of the imager 712 are more accuratelydefined by the filter 1102, thereby making it practical to use a filterhaving finer variations in its filtering characteristics. However, atradeoff for this benefit is a loss of flexibility in changing oraltering the filter which is typically associated with a camera havingan internal, rather than an external, filter.

[0155]FIG. 12 illustrates a camera in which a filter 1202 is locatedinside the camera, but is not in direct contact with the sensor 708. Thearrangement of FIG. 12 enables use of a curved or tilted filter.

[0156] In a camera system having a compound lens, as illustrated in FIG.13, a spatially varying filter 1302 can be placed between elements 1306and 1304 of the compound lens. In the example illustrated in FIG. 13,the objective lens 1306 forms an image of the object, and the additionallens 1304 projects the image onto the detector 708. The filter 1302 ispreferably placed close to the additional lens 1304. Virtually allcamera lenses are compound for correcting aberrations etc. and usuallyhave more than two elements.

[0157] As illustrated in FIG. 14, an imager having a compound lens canalso include a diffuser 1406 which is positioned at the focal plane ofan objective lens 1404. A filter 1402 having a spatially varyingcharacteristic is placed as close as possible to the diffuser 1406. Theobjective lens 1404 forms, on the diffuser 1406, a focused image whichis modified by the spatially varying filter 1402. An additional lens1408 projects the focused image from the diffuser 1406 to the detector708.

[0158] An imager in accordance with the present invention can beconfigured to receive radiation (e.g., light) rays from the scene usinga reflector having a spatially varying reflectance. In such a system, anexample of which is illustrated in FIG. 15, the reflector 1502 havingthe spatially varying reflectance is used to reflect light signals 810,812, and 814 from the scene (i.e., from points A, B, and C) into thecamera aperture 710. If the reflector 1502 having the spatially varyingreflectance is partially transmissive (i.e., transmits some light), thereflector 1502 can be arranged between the camera aperture 710 and areference object 1506 which can be used to calibrate the system or toabsorb spurious reflections from the camera 702. For example, awell-known calibration technique uses a known light source to calibratethe characteristics of the detector in the camera. The arrangement ofFIG. 15 can be viewed as functionally equivalent to a virtual camera1504 which is oriented at an angle different from that of the actualcamera 702, and which effectively receives the light ray bundles 810,812, and 814 directly—i.e., not by reflection.

[0159] An additional advantage of using an externally mounted reflector1502 in the imager 712 is that the reflector 1502 can optionally becurved in order to control the field of view and/or the magnification ofthe imager 712.

[0160]FIG. 16 illustrates an imager 712 having a filter 1602 outside thecamera 702, and an additional filter 1604 inside the camera 102.

[0161]FIG. 17 illustrates an imager 712 having a filter 1702 and areference object 1708 outside the camera 702, an additional filter 1704within the optical components 1710 of the camera 702, and yet anotherfilter 1706 inside the camera 702 and positioned near the detector 708.

[0162] An imager in accordance with the present invention—such as, forexample, the imager 712 illustrated in FIG. 17—can be translated inorder to scan a scene 706, as is illustrated in FIG. 18., As the imager712 translates in the illustrated direction, any particular point in thescene 706—e.g., point A in FIG. 18—is imaged along multiple chief rays804, 806 and 808 in the reference frame of the imager 712. In addition,as illustrated in FIG. 19, the imager 712 need not move in a straightline, but can travel along any path 1902, and furthermore, can besimultaneously translated and rotated. In addition, although the filtersillustrated in FIGS. 7-18 are primarily depicted as varying from top tobottom, a filter 704 in accordance with the present invention can varyin any and all directions. For example, the properties of the filter 704can vary azimuthally, in which case the mosaic procedure can includerotating the imager 712 about its optical axis in order to capture theimages for processing. Furthermore, although a filter 704 can be mountedto the camera in a non-flexing and/or non-moving manner, a filter whichflexes or moves with respect to the camera 702 can also be used. In sucha system, the filter can be flexed or moved between each frame of asequence of images.

[0163] As illustrated in FIGS. 20A and 20B, image mosaicing techniquesin accordance with the present invention can be used to extend both thespatial range (i.e., the width or height of the field of view) and thetotal intensity range (and according, the dynamic range) of the imagemeasurements. FIG. 20A illustrates the difference in the spatial rangesof a single frame 2002 and an image mosaic 2004. The image mosaic 2004has been constructed from multiple frames. FIG. 20B illustrates theenhanced spatial range and enhanced intensity range of an image mosaic2004. In practice, because the pixels near the edges of the mosaic 2004are sampled fewer times, the total intensity range of the edge pixels islikely to be less than the intensity range of pixels in the center ofimage mosaic 2004. This effect can be seen in the graph of FIG. 21,which illustrates that in the regions 2102 near the edges of the spatialrange of the image mosaic 2004, the total intensity range falls off.However, in the portion 2104 which is further away from the edges, theintensity range is uniformly high. This effect is analogous to thefoveated vision of the human eye—which has greater resolution near thecenter of the field of view—and can be advantageous for applications inwhich the center portions of a scene are of greater interest than theedge portions, and are therefore preferably imaged with higherresolution than the edge portions.

[0164] As discussed above, because an externally mounted filter tends tobe out of focus with respect to the detector of a camera, the effect ofsuch a filter upon the camera characteristics tends to be blurred. Forexample, FIG. 22A illustrates a filter density function 2202 in whichthe density D varies linearly across one dimension of the surface of thefilter, between a first position 2214 and a second position 2216, but isconstant outside this range. The transmittance T equals 10^(−D), andtherefore, the corresponding intensity sensitivity characteristicfunction of the resulting imager varies exponentially across the filter.The effective density function 2204 of the filter has rounded cornersdue to blurring of the filter density function 2202. Similarly, asillustrated in FIG. 22B, a filter having a density function 2206 withsteps will have an effective density function 2208 with rounded corners.As illustrated in FIG. 22C, a filter mounted externally to a camera canhave any arbitrary density function 2210, but if the function 2210 hassharp corners, the effective function 2212 will have rounded corners.

[0165] In general, for a spatially varying attenuation mask function f,the effective mask function M can be modeled as:

M(x,y)=f(x,y)*h(x,y)  (25)

[0166] where h(x,y) is the defocus blur point spread function (PSF) ofthe camera for objects as close as the filter, when the system isfocused at the distant scene. For circularly symmetric PSFs' the mask iseffectively a one dimensional function of x, and f is convolved with{tilde over (h)}, the Abbel transform of h. For example, if the kernelis modeled as a Gaussian function, then h is a Gaussian function havinga standard deviation *, and M(x)=erf(−x/r). Since M(x) takes any valuebetween 0 and 1, then in principle any scene point can be imaged withoutsaturation, regardless of how bright the point is, if it is seen throughthe filter at the appropriate location. Therefore, this simple systemtheoretically can have an infinite dynamic range.

[0167] The effect of the blurring of an externally mounted attachmentcan be further illustrated using as an example a simple occluder 2302,as depicted in FIG. 23. Although the occluder 2302 itself clearly has anabrupt spatially varying characteristic, its effect on thecharacteristics of the camera 702 varies more gradually across the fieldof view of the camera 702, as illustrated in FIGS. 24A and 24B. FIG. 24Aillustrates the more rounded effective density function 2402 of thesimple occluder 2302. FIG. 24B illustrates the logarithm of theeffective density function 2406 of the simple occluder 2302. Inaddition, the blurring effect on the effective density function 2404 ofa linearly varying density filter is also illustrated in FIG. 24A. Thelogarithm of 2408 of the effective density function 2404 of the linearlyvarying density filter is illustrated in FIG. 24B. The logarithmicfunction 2408 is a straight line.

[0168] The above-described blurring effect which tends to occur inimagers having externally mounted optical elements can be advantageousfor the adjustability of an imager, because the optical characteristicsof the imager can be altered merely by changing the position of theoptical element, along the optical axis of the imager. For example, theslope near the middle of the effective density function 2402 of a simpleoccluder 2302 (see FIGS. 24A and 23) can be increased by moving theoccluder 2302 closer to the focal distance of the camera 702 anddecreased by moving the occluder 2302 farther away from the focaldistance of the camera 702.

[0169] Although the above discussion has emphasized spatially varying,wavelength-neutral density filters which result in imagers havingspatially varying wavelength-neutral intensity sensitivitycharacteristics, it is also possible to employ a spatially varying colorfilter or other wavelength-based filter. Such a spatially varyingwavelength filter can, for example, be a spatially varying bandpassfilter (e.g., an interference filter) having a spatially varying centralwavelength λ₀. FIG. 25A illustrates a function 2502 of centralwavelength λ₀ versus position x across a filter having a centralwavelength λ₀ which varies linearly across the field of view. The filterfunction 2502 produces an imager having a wavelength sensitivitycharacteristic function 2504 which also varies approximately linearlyacross the field of view, but has rounded corners. FIG. 25B illustratesan example of a filter having a central wavelength characteristicfunction 2506 which varies in steps across the field of view. Such afilter results in an imager having a wavelength sensitivitycharacteristic function 2508 which roughly corresponds to the step-wisefunction 2506 of the filter, but which has rounded corners duetoblurring, because the filter is somewhat out of focus with respect tothe camera. As illustrated in FIG. 25C, a bandpass filter having anyarbitrary central wavelength function 2510 can be used, but if thefunction 2510 of the filter has sharp corners, the resulting imager willhave a wavelength sensitivity characteristic function 2512 with morerounded corners.

[0170] An imaging system in accordance with the present invention canalso employ a spatially varying low pass or high pass filter. Forexample, FIGS. 26A-26C illustrate exemplary functions 2602, 2606, and2610 of cutoff wavelength λ^(cut) versus position x across exemplary lowpass or high pass filters. Similarly to the filters illustrated in FIGS.22A-22C and 25A-25C, blurring causes rounding of the respectivewavelength sensitivity characteristic functions 2604, 2608, and 2612 ofthe imagers formed using the respective filters.

[0171] The blurring effect is further illustrated by FIG. 27 which is agraph of detector sensitivity versus wavelength λ for a light ray bundlehaving a chief ray near the border of two segments of a high pass filterarray. Half of the light in the beam passes through a segment having acutoff wavelength of λ₁^(cut),

[0172] and the other half of the beam passes through a segment having acutoff wavelength of λ₂^(cut).

[0173] The effective cutoff wavelength of the detector isλ_(effective)^(cut),

[0174] which is between λ₁^(cut)  and  λ₂^(cut).

[0175] Compared to the respective transition bands of the two individualsegments, the transition band for the beam having a chief ray on theborder is broader.

[0176] Similarly to spatially varying attenuating filters, discussedabove, the effective characteristics of a wavelength filter can also beadjusted by changing the filter's position along the optical axis of theimager. In particular, the filter's features become sharper, lessrounded, and more highly sloped if the filter is closer to the focaldistance of the camera, and the features become more rounded and lesshighly sloped as the filter is moved away from the focal distance of thecamera. In fact, a step-wise array of wavelength filters—of whichexemplary characteristics 2506 and 2606 are illustrated in FIGS. 25B and26B, respectively—can, if sufficiently out of focus, be used toapproximate a linearly varying filter—of which exemplary characteristics2502 and 2602 are illustrated in FIGS. 25A and 26A, respectively.

[0177] A spatially varying bandpass filter in accordance with thepresent invention can also be used to enhance the precision (i.e., thedynamic range) of a color imager with respect to discrimination amongthe wavelengths of signals received from a scene. As illustrated in FIG.28, a typical color imager (e.g., a color camera) has three relativelybroad color channels 2802, 2804, and 2806, receiving blue, green, andred light, respectively. By attaching to the imager a filter having arelatively narrower pass band 2808, the imager's ability to discriminateamong wavelengths is enhanced. For example, if the pass band 2808 of thenarrow band filter is within the broader blue band 2802 of the camera,the blue light 2810 ultimately detected by the camera is much strongerthan the green and red light 2812 and 2814. Accordingly, because thecharacteristic 2808 of the narrow hand filter is not as broad as thecharacteristics 2802, 2804 and 2806 of the original camera, color isdetermined with finer detail than merely designating a signal as “blue.”The central wavelength λ₀ of the pass band 2808 of the narrow bandfilter varies across the field of view, and therefore, by panning theimager across the scene and capturing multiple images, each point of thescene is imaged within multiple color bands, each having a differentcentral wavelength λ₀. As a result, the spectral characteristics oflight emanating from each point of the scene are determined with greatercolor resolution.

[0178]FIG. 29 illustrates the use of an external color filter dividedinto three large portions, a blue filter portion having a blue bandpasscharacteristic 2904, a green filter portion having a green bandpasscharacteristic 2906, and a red filter portion having a red bandpasscharacteristic 2908. The three-color filter array is mounted externallyto a color camera having its own set of three-color channels, as definedby the band pass characteristics 2902, 2912, and 2910 of the blue, greenand red light, respectively, passed through to the detector in thecamera. As the imager pans across a scene, each portion of the scene isviewed and imaged through the external blue filter portion, the externalgreen filter portion, and the external red filter portion. As a result,each of the blue, green, and red characteristics 2904, 2906, and 2908 ofthe respective filter portions is used in combination with each of theblue, green, and red characteristics, 2902, 2912, and 2910 of the cameraitself. The imager therefore effectively has nine color bands,representing all of the possible combinations of sensor channels andexternal filter element characteristics. Accordingly, the colorresolution of the camera is enhanced. In general, if the camera itselfhas N_(s) bands, and the additional mask has N_(m) bands, then theimager as a whole can measure N_(s)N_(m) bands.

[0179] In accordance with an additional aspect of the present invention,an imager having spatially varying focal characteristics can be pannedacross a scene, and the distances of scene points from the imager 712can be determined based upon how well focused each scene point is ineach image of a set of images captured during the panning of imager 712.FIG. 31 illustrates the response of an imager 712 to light ray bundles3108 and 3110 emanating from an in-focus portion 3104 of a scene and adefocused portion 3106 of a scene, respectively. An objective lens 3102is positioned at a distance v from the focal plane 3118 of a camera 702.The in-focus scene point 3104 is projected as a single point 3114 ontothe focal plane 3118. However, the defocused scene point 3106 isprojected onto the focal plane 3118 as a disk 3116 having a diameter d.

[0180] In general, objects at distance ũ from the aperture 710 of thecamera 702 are in focus, and accordingly, any object which appears to bein focus is at distance ũ. In contrast, any object which appears to beout of focus is known to be at a different distance (e.g., u) from theaperture 710.

[0181] An imager can be configured such that the distance u at whichobjects are in focus varies across the field of view of the imager. Ifthe variation of the focal characteristic is sufficiently large, andenough snapshots are taken, then all or most of the points in the scenewill, in at least one of these snapshots, be in focus or nearly infocus. By determining the particular chief ray received by the imageralong which the object is in focus, and by knowing the spatially varyingfocal characteristics of the imager, the distance of every scene pointcan be determined by the following procedure.

[0182] Any optical system has a range of distances at which objects arein focus or nearly in focus, while becoming blurred as they move awayfrom the range. The size of this range is commonly referred to as the“depth of field” of the system. An imaging system in accordance with thepresent invention converts this range of distances/depths to a range oflocations within the imager- along the optical axis of the imager. Thesize of this range of locations can be referred to as the “depth offocus.” In many applications the range of depths in which scene objectsreside is wider than the depth of field, and therefore, some objectswill be defocused. Referring to the imaging system depicted in FIG. 31,the blur diameter d of an object point at distance u is: $\begin{matrix}{{d = {D\frac{\left| {{uF} - {\overset{\sim}{v}u} + {F\overset{\sim}{v}}} \right|}{Fu}}},} & (26)\end{matrix}$

[0183] where F the focal length and D is the aperture of the lens. It ispossible to capture multiple images of the object with different focussettings (e.g., by moving internal parts of the system such as the lensor the detector during acquisition). The focused state of an objectpoint is usually detected by selecting the image which maximizes imagesharpness at the image point corresponding to the object point.Furthermore, a completely sharp image of the entire scene can beconstructed using a “fusion” procedure. Such a procedure selects, foreach scene region or scene point, the corresponding point in the imagein which the scene region or point is sharpest. The entire set ofselected points is then combined to form a sharp image of all or most ofthe scene points.

[0184] Other techniques can also be used to measure the distances/depthsof scene points. For example, if the focus settings are changed bychanging {tilde over (v)}, and sharpness is measured by the imageLaplacian, then the place v of best focus can be estimated as:$\begin{matrix}{v = {\arg \quad \max {{\nabla_{x,y}^{2}{I_{\overset{\sim}{v}}\left( {x,y} \right)}}}}} & (27)\end{matrix}$

[0185] where ∇_(x,y) ² denotes the Laplacian in the image plane, andeach image I (x,y) is parameterized by {tilde over (v)}—i.e., defined asa function of {tilde over (v)}. Note that once the plane of best focusis determined, it can be used to infer the depth u of the object point.Moreover, the depth can also be inferred not only by seeking the planeof best focus, but by estimating the blur diameter d using a methodcalled “depth from defocus” based on two or more frames. The “depth fromdefocus” technique is discussed in detail in U.S. Pat. No. 6,229,913,entitled “Apparatus and Methods for Determining theThree-DimensionalShape of an Object Using Active Illumination and Relative Blurring inTwo-Images Due to Defocus,” issued on May 8, 2001 to Nayar et al., whichis incorporated herein by reference in its entirety.

[0186] A scene point is represented, in an image, by a region having asize corresponding to a position uncertainly U (i.e., an amount of blur)which can be due to aberration, diffraction blur, or the finite distanceΔx_(grid) between adjacent elements of the detector array. If the blurdiameter d is no larger than U, the scene point is considered to bewithin the depth of field of the system and is thus considered to befocused. Because the blur diameter depends on the distance from thestate of best focus, a change of U in d corresponds to a particularchange in depth (i.e., distance). Consider, for example, a case in whicha component of the imaging system is moved by an amount Δv in order tocapture a set of images over a particular interval of depth of field.Instead of physically changing Δv, the optical path length can bechanged using a piece of transparent dielectric having a refractiveindex n. To make the adjustment, the thickness t or refractive index nof the dielectric should change. In fact, both approaches can be appliedtogether. According to the rate at which the thickness or refractiveindex changes across the field of view, the system calculates theinter-frame transversal increment which yields the desired, effectiveaxial change over the selected interval of depth of field.

[0187] It is possible to avoid moving parts by moving the entire opticalsystem closer to, or farther from, the scene objects. However, such anapproach is practical only if the range of depths is very small, such asin microscopy applications. On the other hand, if a piece of transparentdielectric such as glass or polycarbonate is placed on the detector, theoptical path length between the detector and the lens(es) becomeslonger. For a material having an index of refraction n, each millimeterof material along a light ray is equivalent to propagation through nmillimeters free space. This effectively elongates a portion of {tildeover (v)} by a factor of n (if the refractive element is internal to thecamera) or elongates a portion of ũ (if the refractive element isexternal to the camera). Thus, object points which had been focusedbecome defocused. On the other hand, some object points which wouldotherwise be focused on a plane behind the detector plane now becomefocused on the detector.

[0188] An example of a spatially varying, refractive element inaccordance wiih the invention is a piece of transparent material havinga thickness that varies across the field of view. Alternatively, or inaddition, to the spatially varying thickness, the filter can have aspatially varying refractive index. Examples of the characteristics ofsuch filters are shown in FIGS. 32A-32C. FIG. 32A illustrates therefractive properties of a prism or a refractive element having alinearly varying index. Such an element, if placed outside the opticalsystem—such as outside the imager 712 illustrated in FIG. 8—willprimarily only deflect the field of view, thereby changing the center ofprojection of the system, and will aid little in extending the depth offield of the imager. Therefore, it is preferable to position thiselement on or near the detector—such as in the imager 712 illustrated inFIG. 11. If an intermediate image is formed on a diffuser 1406—such asin the imager 712 illustrated in FIG. 14—then the element 1402 ispreferably positioned immediately before the diffuser.

[0189] If the refractive properties and/or the thickness of therefractive element vary as a function of position within the field ofview, then the resulting imager will have a spatially varying focalcharacteristic. In the example in FIG. 32B, the thickness or refractiveindex of the refractive element varies in a step-wise fashion. In fact,the refractive element can have a thickness or refractive index whichvaries according to any arbitrary function, as illustrated in FIG. 32C.

[0190] The foregoing discussion has described the use of imagers whichhave sensitivity of focal characteristics which vary across the field ofview. As discussed above, such spatially varying sensitivity or focalcharacteristics can include a spatially varying intensity sensitivitycharacteristic (e.g., resulting from a spatially varying attenuator), aspatially varying wavelength sensitivity characteristic (e.g., resultingfrom a spatially varying color filter), a spatially varying polarizationsensitivity characteristic (e.g., resulting from a spatially varyingpolarizing filter), and/or a spatially varying focal characteristic(e.g., resulting from a spatially varying refractive element). Inaccordance with an additional aspect of the invention, two or more ofthe aforementioned properties can be combined in a single imager. Forexample, as illustrated in FIG. 33, some or all of a spatially varyingattenuator 3302, a spatially varying spectral filter 3304, a spatiallyvarying polarizing filter 3306, and a spatially varying refractiveelement 3308 can be simultaneously attached internally or externally toa camera, thereby producing an imager having multiple characteristicswhich vary across the field of view of the imager.

[0191] In some cases, each characteristic of light is independent of theother characteristics. For example, in volumetric objects each imagepoint receives light from multiple (or a continuum of) points havingdifferent depths. The light emanating from any of these points caninclude light which originated from different points or regions of thescene (e.g., scattering or emission from different points or regions ofthe scene). As a result, each spectral component may have a differentpolarization, a different brightness and a different state of focus.

[0192] However, in most cases, some of the characteristics can bedegenerate or highly coupled. For example, some of the radiation whichcomes from a specific depth—within a specific polarization component andspectral band—may be too strong to be detected without saturation, andtherefore, the intensity order of magnitude should preferably be sensedwith a wide brightness dynamic range filter. In addition, if the objectis not three-dimensional then the depth measurement should be doneseparately from the spectral and polarization measurements. Inparticular, to avoid errors of depth estimation caused by detectorsaturation, it may be beneficial to extend the brightness dynamic rangeby using variable density filtering within a filter having a variablethickness or variable refractive index. In such cases, multipleindependent measurements should be preferably performed at each pixelposition, using all of the possible combinations of bandwidthsensitivity, polarization angle sensitivity, and intensity sensitivity.Such measurements can be performed by an imager configured to use acompound filter having various regions, each region being dedicated tothe spatial variation of a particular sensitivity characteristic. Forexample, as illustrated in FIG. 34, a filter 3410 having a first segment3402 for spatially varying a first sensitivity characteristic of theimager, a second segment 3404 for varying a second sensitivitycharacteristic, a third segment 3406 for varying a third sensitivitycharacteristic, and a fourth segment 3408 for varying a fourthsensitivity characteristic may be used. In addition, the varioussegments 3402, 3404, 3406, and 3408 need not be defined to have anyparticular shape or location, or have their respective filtercharacteristics spatially varying in any particular direction, but canbe regions of any shape and/or location, and filter characteristicsspatially varying in any direction on the filter 3410.

[0193] It is to be noted that the variation of each filtercharacteristic can be configured to have a different spatial frequencyon the filter. For example, the depth can vary as cos x, the neutraldensity can vary as cos 2x, and the central wavelength can vary as cos4x. Furthermore, the spatial variation of each filter characteristicneed not be sinusoidal. Square wave patterns, saw-tooth patterns, andother patterns can also be used.

[0194] On the other hand, certain filter characteristics may not requirea separate filter region dedicated to them. For example, thepolarization of light coming from a scene point or region will usuallynot depend strongly on the wavelength of the light. Accordingly aspatially varying polarizing filter can overlap a spatially varyingwavelength filter with little or no loss of information. Furthermore,light from all parts of the scene need not be measured with respect toits characteristics, or with the same resolution. In addition, forapplications in which speed is more important than completeness of themeasurement, a faster scan can be obtained using a set of overlappingfilters such as the filters 3302, 3304, 3306, and 3308 illustrated inFIG. 33, rather than the compound filter 3410 illustrated in FIG. 34;the compound filter 3410 typically requires a slower scan to obtain agiven resolution.

[0195] Once images have been acquired by a scanning procedure inaccordance with the present invention, the data acquired atcorresponding pixel positions—i.e., pixel positions representing thesame scene point—are processed in order to extract the values of highdynamic range intensity, polarization state, depth, and color at eachpixel of a processed image. The sequence of images are registered inorder to enable fusion of all the raw data gathered for each scenepoint.

[0196]FIG. 35A illustrates a sequence of snapshots 3502, 3504, 3506,3508, and 3510 in which light from a scene point is detected at pixelpositions 3522, 3524, 3526, 3528, and 3530 in the respective snapshots3502, 3504, 3506, 3508, and 3510. By determining the position ofwhichever pixel represents the scene point in each snapshot, thesnapshots 3502, 3504, 3506, 3508, and 3510 can be mathematically alignedas illustrated in FIG. 35B, and the data corresponding to the respectivepixel positions 3522, 3524, 3526, 3528, and 3530 can be processed tothereby generate an improved quality pixel having enhanced dynamicrange, spectral information, polarization information, and/or depthinformation, as discussed above with respect to various image mosaicingtechniques in accordance with the present invention. The registrationprocedure can include one or more of the following approaches:

[0197] 1. The motion between frames may be known. This is the case whenthe imaging system is mounted on a motor and its speed is calibrated interms of the shift in viewing direction between successive images. Thismay also be the case for images taken from a satellite, plane, or othervehicle whose position is monitored.

[0198] 2. The images can be manually registered, e.g., by matchingcontrol patches in the images. These patches usually contain prominentimage features whose morphology is robust to changes in thecharacteristics of light from such image features. In other words, theshape of a control patch is similar in different wavelength bands,polarization states, and a broad range of light intensity distributions.

[0199] 3. The images can be automatically registered by tracking thecontrol patches described in item 2. Such registration procedures arewell known. Some methods use higher level descriptions of the features.However, most approaches maximize the correlation or minimize thedifference between the patches. Such methods typically perform better ifthe change undergone by the image between frames is small. Otherwise,the similarity measure may fail. For example, the content of the sceneat λ=630 nm may be totally unrelated to the content at λ=400 nm.However, if the images are captured using wavelengths which do notdiffer by large amounts, and the wavelength bands of the light used tocapture the images are broad enough, then consecutive images willusually have sufficient similarities to enable them to be matched.

[0200] 4. The images can be automatically registered by finding thewarping parameters, among the images, which provide the best globalmatch. Such approaches are well known. Commonly used criteria formatching include quality variants of the MSE criterion, robuststatistics error criteria, and correlation. Such methods are similar tothose in item 3; in that they typically perform better if the chargeundergone by the image between frames is small. Such approaches provideefficient coarse-to-fine comparison and registration of the images.

[0201] 5. When the change undergone by the image between frames islarge, one may view the matching problem as similar to that encounteredwhen registering images taken with different sensors. Although, betweenframes, the scene appears to move through the field of view of theimager, the mask characteristically tends to dominate the images,thereby tending to cause such algorithms to register an imagetranslation which is smaller than the correct one, or even to registerno translation. It is well known that a possible solution to theforegoing problem is to combine the methods of items 3 and 4. In such acombined method, each image is high-pass filtered to enhance thereliance on prominent features which are more likely to be invariant inthe images. The registration is performed using a coarse-to-fineprocedure, while optimizing the global warping parameters between theimages.

[0202] 6. Another general way to register images even if the change islarge, such as in multi-sensor systems, is to use mutual information-i.e., similarity of features between images -as a criterion formatching. This method generally requires that the images are notcompletely independent statistically, which is usually a validassumption.

[0203] In some cases, it is possible to partly compensate for the effectof the filter in order to facilitate matching. For example, suppose animager employs a varying density filter. Such a filter acts as a maskwhich attenuates each part of the field of view by a factor M whichvaries across the field of view. Since the mask is multiplicative inthis case, it can be beneficial to calculate the logarithm of eachimage, in order to make the mask function M an additive component. Then,the effect of the mask can be reduced by high-pass filtering theresulting logarithmic images, because the mask is usually slowlyvarying. If an estimate of M is available before the registrationprocedure begins, then before starting the procedure, each pixel valueof each image can be amplified by the corresponding value of 1/M. Thisstep will alleviate biasing effects on motion estimation, which can becaused by a temporally-constant mask. Measurements which became darkerdue to strong attenuation will tend to be relatively noisy after beingquantized by the sensor. It is preferable to take into account thisattenuation-dependent uncertainty when the images are compared andregistered.

[0204] Note also that the registration can be done with subpixelaccuracy if overlapping areas are sampled more densely than the pixelgrids of the individual images. Therefore, spatial super-resolution canbe obtained concurrently with, or in addition to, the derivation of themultidimensional properties of the scene points.

[0205] An exemplary algorithm is based on a rough estimate of the maskM, with uncertainty ΔM. An estimate of 1/M is used to flat field theimages. As discussed above, pixels which have been attenuated to be verydark (especially those which are quantized to zero) become noisy due tothe map inversion. Therefore, in all the subsequent algorithms, it ispreferable to account for the uncertainty of the measurements when theyare compared, differentiated, and fused. The method is based on thefollowing principles:

[0206] 1. Crude flat fielding by 1/M. Let the light intensity falling onthe detector when M=1 be I. It is measured in units of I_(min)^(detector), making it dimensionless. The light falling on the detectorafter the optical filtering is g(x,y)=M(x,y)I(xy). For each measurementg±Δg the scene irradiance I±ΔI is estimated. The uncertainties arepropagated using first-order approximations.

[0207] 2. Since high dynamic range images are used, errors in dark scenepoints are considered as significant as bright ones. For example, a 5%error in the bright points should be penalized as much an equivalenterror in the dark points. In order to adjust for the brightness of eachpixel, the log of each measured scene radiance is calculated—i.e., s=logI—to yield s±Δs.

[0208] 3. Coarse-to-fine paradigm. As is well known, reliable andefficient approach to registration is to use a pyramid(multi-resolution) representation of the images. After registering andwarping the images at a coarse spatial resolution level, the coarsemotion estimate serves as an initial condition to improve thetransformation estimation at a finer scale, etc. However, in contrast toconventional methods which have ignored data uncertainties, theprocedure of the present invention creates a Maximum-Likelihood Pyramidof the images of s; at each level of the pyramid the value representinga certain neighborhood is the most likely value, considering theuncertainties of the contributing pixels.

[0209] 4. Maximizing the likelihood of the registration. Themeasurements s are estimated to have Gaussian distributions (althoughthe method is not limited to Gaussian distributions). The bestregistration is the one which minimizes the Mahalanobis distance betweenthe images.

[0210] 5. Registration to the mosaic. Registering only pairs of imagesleads to an accumulation of errors of the estimated image positions inthe global coordinate system. Therefore, each new image in the sequenceis preferably registered to the latest updated mosaic, and is then fusedto the updated mosaic.

[0211] The algorithm is explained in further detail as follows. Thedetector has a response R (which may be nonlinear) within its dynamicrange, so that the image value {tilde over (g)} at the pixel is:

{tilde over (g)}=R(g).  (28)

[0212] Linearizing the response, the estimated intensity ĝ at thedetector is

ĝ=R ⁻¹({tilde over (g)})  (29)

[0213] and thus: $\begin{matrix}{{\Delta \hat{g}} = {{\frac{R^{- 1}}{\overset{\sim}{g}}}\Delta {\overset{\sim}{g}.}}} & (30)\end{matrix}$

[0214] Assuming the estimated mask to be independent of the measuredsignal,

(ΔI)²=(Δĝ/M)²+(ĝΔM/M ²)²,  (31)

[0215] where it is assumed that Δĝ=0.5, because the output of the camerais in the form of integers (0-255 for an 8-bit detector). To avoidpotential instability of the log operation at dark points (where I=0),dark points can be eliminated from the subsequent calculations.Therefore, it is preferable to use s=log (1+I), rather than I, for thecalculation. In any case, $\begin{matrix}{{\Delta \quad s} = {{\frac{s}{t}}\Delta \quad {I.}}} & (32)\end{matrix}$

[0216] Note that if the response R of the detector is logarithmic as,there is no need to use Eqs. (29)-(32); s can be set equal to ĝ. Anyimage pixel considered to be saturated (e.g., ĝ close to 255 for an 8bit detector), is treated as being very uncertain. Thus, itscorresponding Δs is set to be a very large number (i.e., out of thesystem's dynamic range).

[0217] The measurements of s are assumed to be Gaussian. Therefore, fortwo independent measurements s₁±Δs₁ and s₂±Δs₂, the log-likelihood for avalue se behaves like as follows: $\begin{matrix}{{\log \quad {\left. {Likelihood} \right.\sim{- E^{2}}}} \equiv {{- \left( \frac{s_{e} - s_{1}}{\Delta \quad s_{1}} \right)^{2}} - {\left( \frac{s_{e} - s_{2}}{\Delta \quad s_{2}} \right)^{2}.}}} & (33)\end{matrix}$

[0218] The maximum likelihood solution for se is the one which minimizesthe Mahalanobis distance E: $\begin{matrix}{{S_{e} = {\hat{\Delta}\quad {s^{2}\left( {\frac{s_{1}^{2}}{\Delta \quad s_{1}} + \frac{s_{2}^{2}}{\Delta \quad s_{2}}} \right)}}},{{where}\text{:}}} & (34) \\{{\hat{\Delta}\quad s^{2}} = {\left( {0.5*\frac{^{2}E^{2}}{S_{e}^{2}}} \right)^{- 1} = {\left( {\frac{1}{\Delta \quad s_{1}^{2}} + \frac{1}{\Delta \quad s_{2}^{2}}} \right).}}} & (35)\end{matrix}$

[0219] Then, the distance values for the image measurements thatcorrespond to this scene point is: $\begin{matrix}{{\hat{E}}^{2} = {\left( \frac{\hat{s} - s_{1}}{\Delta \quad s_{1}} \right)^{2} + {\left( \frac{\hat{s} - s_{2}}{\Delta \quad s_{2}} \right)^{2}.}}} & (36)\end{matrix}$

[0220] Assuming all pixels in the image to be independent, the distancemeasure Ê_(total) for the entire frames, or for any other subgroup ofpixels is: $\begin{matrix}{{\hat{E}}_{total}^{2} = {\sum\limits_{{all}\quad {pixels}}{{{\hat{E}}^{2}\left( {{each}\quad {pixel}} \right)}.}}} & (37)\end{matrix}$

[0221] The best registration between two frames (or between a new frameand an existing mosaic) according to the above objective function is theone which minimizes Ê_(totol). Here, each pair of measurements s₁ and s₂corresponds to the values of the images at the corresponding pixels.

[0222] Note that Ê_(total) will generally increase with the number ofpixels. This may bias the registration towards minimizing the number ofpixels in the sum in Eq. (37), thus reducing the overlap between theimages. To counter this effect, Eq. (37) can be normalized by the numberof pixels in the overlap area, or by Δs_(total) ⁻¹, or Δs_(total) ⁻²,etc., where: $\begin{matrix}{{\Delta \quad s_{total}^{2}} \equiv {\left( {\sum\limits_{{all}\quad {pixels}}{\hat{\Delta}\quad s_{{each}\quad {pixel}}^{- 2}}} \right)^{- 1}.}} & (38)\end{matrix}$

[0223] If the statistical dependence between different measurements orbetween different pixels cannot be neglected, then the equations forÊ²  and  Ê_(total)²

[0224] can be generalized to use the covariance matrix of themeasurements, rather than just their variances.

[0225] To make the registration more robust and efficient, it is donehierarchically, from coarse to fine resolution. A coarse representationof an image at a specific pyramid level can be obtained by sub-samplingthe image after lowpass filtering it with a kernel having a width whichdepends on the level (the higher/coarser the level, the wider is thekernel which operates on the original image). In any case, the value ofa pixel in this representation is a weighted and normalized sum of themeasured pixel: $\begin{matrix}{s = {\frac{\sum\limits_{k}{\omega_{k}s_{k}}}{\sum\limits_{k}\omega_{k}}.}} & (39)\end{matrix}$

[0226] where ω_(k) is the weight for pixel value S_(k).

[0227] It is to be noted that the discussion herein refers to theconstruction of the pyramid levels from the original, full resolutionimage, where the pixels may be considered as independent. This is doneto keep the derivation simple. However, usually pyramids are constructediteratively, in which case the pixels in the intermediate levels are notstatistically independent. If additional accuracy is sought in theiterative process, the weighting should rely not only on the pixelvariance in the intermediate levels, but on their full covariance matrixwith their neighbors. This matrix should be thus propagated up thepyramid as well.

[0228] In a conventional pyramid, the weights ω_(k) are equal to valuesof a Gaussian kernel α_(k). However, in the method of the presentinvention, Eq. (39) can be viewed as a generalization of Eq. (34). Theweight ω_(k) that should be assigned to a pixel linearly decreases bothas its Gaussian weight α_(k) decreases, and as its uncertainty ΔS_(k)increases. Thus ω_(k)=α_(k)/Δs_(k) ², and: $\begin{matrix}{{\Delta \quad s} = {\left( {\sum\limits_{k}^{\quad}\quad \omega_{k}} \right)^{- 1}.}} & (40)\end{matrix}$

[0229] In the above-described Maximum Likelihood pyramid, therepresentation at each resolution level includes not only theweighted-averaged value at each pixel, but also the uncertainty of thevalue of each pixel. Since points having smaller uncertainties receivemore weight in this pyramid, the uncertain areas (such as the saturatedregions or very dark points) have reduced importance at the coarserlevels, and their representation is more influenced by adjacent stablepoints. The representation of a region by one value is thus made morereliable. Accordingly, using having a representation of s±Δs at eachscale enables efficient registration of the images by maximizing thelikelihood of the match.

[0230] It is to be noted that the above-described procedure is not theonly way to register and fuse an image sequence. For example, otherweights can be used for determining the value at each point, or a singlegood measurement of each point can be used to represent the point. Ifthe motion field is parameterized, the neutral density mask does notchange over time, and the scene is static, then all the parameters ofthe mask, the motion, and the unknown scene can be posed as a singleoptimization problem.

[0231] Once the set of images have been captured and registered, and theimager characteristics are known (e.g., by calibration), an algorithm inaccordance with the present invention generates at least one datum(e.g., a pixel) to represent each point or small region of the scene.Such a fusion procedure generates the representative pixel using theparticular pixel, from each of the original images, which corresponds tothe scene point being represented. The choice of fusion proceduretypically depends upon which scene characteristic—e.g., brightness,color, polarization, or depth -is being imaged.

[0232] For example, to fuse intensity (i.e., brightness) images takenwith spatially varying

[0233] 1. The value selected for the point is the one that is mostlikely - hence minimizing the Mahalanobis distance to the data points.Thus Eqs. (34) and (35) are used with s_(k)=I_(k), where I_(k)=ĝ_(k)/M.

[0234] 2. Saturated points in the raw images (ĝ close to 255 in an 8-bitcamera) are assigned a very large uncertainty, as described above.

[0235] 3.A seam minimization procedure is used to avoid aestheticallyunappealing seams which may appear on the image boundaries. On theboundaries of the images that form the mosaic there are transitionsbetween points which have been estimated using different sources ofdata. There are numerous ways to remove seams from mosaics. One approachis to search for an “optimal” seam line—i.e., the seam line whichappears least obvious to the observer. Optimal seam line selectionprocedures are widely used in the art. Another approach is to usefeathering procedure in which the images are weighted according to thepixel position with respect to the image centers or boundaries. Thisweighting fits easily into the weighted averaging described in item 1,and is similar to the above-described use of weights in the constructionof pyramids. For example, the uncertainty can be multiplied by a factorthat smoothly increases to m near the image boundaries.

[0236] 4. Seams also typically appear at the boundaries of saturatedareas, where there is an abrupt change in the uncertainty, while thechange in {tilde over (g)} is usually small. Such seams can be removedusing an approach such as that discussed in item 3 above. It ispreferable to smooth the definition of saturated areas using thefollowing procedure. A “low saturation value” a is defined (e.g., 230for an 8-bit camera). Then a “high saturation value” b (e.g., 250) isdefined. If {tilde over (g)}>b, the pixel is considered to be saturated.If b≧{tilde over (g)}>a and the point is adjacent to a saturated point,the point is considered to be “saturation-associated”. If b≧{tilde over(g)}>a and the point is adjacent to a “saturation-associated” point, itis also considered as “saturation-associated”. Points which are“saturation-associated” are thus always included in groups which areconnected to a saturated point. Therefore, the procedure does not affectintense {tilde over (g)} points ({tilde over (g)}>a) which are notrelated to saturated points. After all the points which have beendeclared as “saturation-associated” have been found, their uncertaintyis multiplied by ≈(b−a)/(b−{tilde over (g)}), or some other functionthat gradually makes the transition from a regular uncertainty(multiplied by a factor of 1) to a very large uncertainty as thesaturated value is reached.

[0237] It is to be noted that fusion of data in accordance with theinvention is not limited to the above-described, exemplary procedures.Data can also be fused using methods of robust statistics to reduce theeffects of outliers, or using iterative projection methods. Seam removalcan also be done in a multi-resolution approach—i.e., by analyzing theimage at several different resolutions, as is well-known in the art.

[0238] It is also to be noted that guidelines 3 and 4 above are notessential for the production of the high dynamic range mosaic, but aremerely used to make the resulting image more aesthetically pleasing.Additional methods include: (1) selecting only a single “good”measurement of each point as the representative value of the point; and(2) selecting the values which yield the maximum contrast (e.g., inmultiple scales [Explain]).

[0239] In the case of polarization data, the image points are analyzedas follows. Assuming that the elliptic polarization component can beneglected, and using three measurements for each point, Eq. (24) is usedto derive the components of the polarization state for each pixel. Ifmore than the minimum required number of images is used, then anoverdetermined set of equations can be used. The set of equations issolved by minimizing the square error, leading to $\begin{matrix}{\begin{bmatrix}C \\A_{c} \\A_{s}\end{bmatrix} = {\left( {M^{\prime}M} \right)^{- 1}M^{\prime}{\overset{\_}{g}\quad.}}} & (41)\end{matrix}$

[0240] Then, the procedure sets I=2C, A={square root}{square root over(A_(c) ²+A_(s) ²,)}P=A/C and θ=sin⁻¹(A_(s)/A)=cos⁻¹(A_(c)/A).

[0241] It is to be noted that the above-described procedure is not theonly way in which the data can be used to extract the polarizationinformation. For example, robust statistics can be used to estimatepolarization. P, θ, and I can be estimated directly—i.e., by passing theintermediate variables A_(c), A_(s), and C; projection iterative methods(e.g., enforcing the positivity of I and P) can be used; or it may besufficient not to extract the polarization information, but merely tofuse the images in a way that will yield the maximum contrast, bycomparing the Laplacian pyramid representations of the images andselecting the representation having the maximum value, as is well-knownin the art.

[0242] To derive depth information, a depth from focus procedure isperformed on the corresponding points, selecting the images that givethe best focus, as done in conventional depth from focus techniques.Then, the selected images can be fused so that only “sharp” pixelscontribute to the mosaic. For example, if the Laplacian measures theimage sharpness, then:

Î(x,y)=I _(p(x,y))(x,y) such that p(x,y)=arg max|∇²_(x,y)I_({tilde over (p)})(x,y)|,  (42)

[0243] where (x,y) are the coordinates in the stationary (outside world)coordinate system, and {tilde over (p)} is the index of the frame.However, the fusion need not be only at the pixel level, but can be donein multiple resolution scales. Moreover, the technique is not limited tothe Laplacian criterion for focus; other focus criteria can also beapplied.

[0244] To derive the spectrum, corresponding measurements of the samescene point in different bands are concatenated. If the filter is not abandpass filter, but a highpass or lowpass filter, then the spectralinformation in narrow bands can be derived by differentiation ofadjacent measurements. The spectrum can be measured using bands ofvarying widths, and can even be composed of multiple bands. The highresolution spectrum is extracted by solving the set of equationsprovided by the raw data, considering the structure of the bands, thepositivity of the spectrum, and other constrants. Moreover, it ispossible to use a priori information regarding the spectra of knownsources and materials to determine which types of known sources andillumination components are consistent with the measurements, includingthe relative intensities, temperatures, and/or spatial locations of themeasurements.

[0245] Using the methods of the invention described herein, the dynamicrange of any video camera (or still camera) can be extended, thespectral resolution of the camera can be greatly enhanced, and/or thepolarization states and focal properties of incoming signals can beresolved simply by attaching a filter in front of the lens of thecamera, or even by exploiting pre-existing vignetting characteristics ofthe camera.

[0246] Knowledge of the spectral content of a scene can be used to inferinformation regarding the light sources illuminating the scene at thetime the photographs were taken, and can also be used to infer objectreflectance. As a result, images of the scene can be rendered usingdifferent simulated illumination sources and characteristics. Forexample, an object photographed under an incandescent lamp can berendered, in a simulated manner, as if photographed under fluorescentlight, or under a sunset-lit sky.

[0247] Furthermore, it is to be noted that the spectral response of mostdetectors is very different from that of the human eye, even when RGBfilters are used in a detector. Film and color prints suffer fromsimilar limitations. Knowing the actual spectral content of the scenethus enables rendering of the images in a manner closer to what thehuman eye would detect if viewing the scene directly. Images can also berendered to be consistent with the responses of films, screens, orprinters, rather than typical eye response.

[0248] The techniques of the present invention extend the opticaldynamic range of an imager while creating mosaics. The dynamic range isextended for each point in the scene, regardless of the brightness ofsurrounding points. The imager need not have any internally movingparts, and the motion of the imager is the same as that required tocreate an ordinary image mosaic. The techniques of the invention canalso be combined with other methods for enhancing dynamic range, such asactive CMOS detectors or AGC, to thereby provide further enhancement.Note that the optical dynamic range is larger if the density filter hasa wider range of density. For example, if Mε[10⁻⁴, 1] the dynamic rangeof the detector is enhanced by approximately 13 bits beyond itsintrinsic range (equivalent to an 80 dB improvement). Putting severalsuch filters in series enhances the spatially varying attenuationeffect, since filter density is an additive property.

[0249] Enhancing the depth of focus enables the creation of sharp imagesover wide distance ranges. Acquisition of polarization data enablesremoval or enhancement of reflection and semi-reflection visual effects.

[0250] It will be appreciated by those skilled in the art that themethods illustrated in FIGS. 1-6, 35A, and 35B can be implemented onvarious standard computer platforms operating under the control ofsuitable software as exemplified by the programs in the Appendix. Insome cases, dedicated computer hardware, such as a peripheral card whichresides on the bus of a standard personal computer or workstation, canenhance the operational efficiency of the above methods.

[0251]FIGS. 37 and 38 illustrate exemplary computer hardware suitablefor practicing the present invention. Referring to FIG. 37, the computersystem includes a processor section 3710, a display 3720, a keyboard3730, and a mouse. The system can also include other input devices suchas an optical scanner 3750 for scanning an image medium 3700, and acamera 3780. In addition, the system can include a printer 3760. Thecomputer system typically includes one or more disk drives 3770 whichcan read and write to computer readable media such as magnetic media(e.g., diskettes), or optical media (i.e., CD-ROMS), for storing dataand application software.

[0252]FIG. 38 is a functional block diagram which further illustratesthe processor section 3710. The processor section 3710 generallyincludes a processing unit 3810, control logic 3820 and a memory unit3830 and a hard disk drive and interface. The processor section 3710further includes a timer 3850 (i.e., a clock circuit) and input/outputports 3840. The processor section 3710 can also include a co-processor3860, depending on the microprocessor used in the processing unit.Control logic 3820 provides, in conjunction with processing unit 3810,the control necessary to handle communications between memory unit 3830and input/output ports 3840. Timer 3850 provides a timing referencesignal for processing unit 3810 and control logic 3820. Co-processor3860 provides an enhanced ability to perform complex computations inreal time. Present day computers are more likely to have multipleprocessing units than a co-processor.

[0253] Memory unit 3830 can include different types of memory, such asvolatile and non-volatile memory and read-only and programmable memory.For example, as shown in FIG. 38, memory unit 3830 can include read-onlymemory (ROM) 3831, electrically erasable programmable read-only memory(EEPROM) 3832, and random-access memory (RAM) 3833. Different computerprocessors, memory configurations, data structures and the like can beused to practice the present invention, and the invention is not limitedto a specific platform. For example, although the processor section 3710is illustrated as part of a computer system in FIG. 37, the processorsection 3710 and/or the illustrated components thereof can also beincluded in an imager such as a still-image camera or a moving-imagecamera (e.g., a video camera).

[0254] Software exemplified by the source code listing in the Appendixcan be written in a wide variety of programming languages, as will beappreciated by those skilled in the art. Exemplary software algorithmsin accordance with the present invention have been written in theMatlab™ language. The computer source code for several of theseexemplary algorithms is provided in the Appendix attached hereto.

[0255] Although the present invention has been described in connectionwith specific exemplary embodiments, it should be understood thatvarious changes, substitutions, and alterations can be made to thedisclosed embodiments without departing from the spirit and scope of theinvention as set forth in the appended claims.

We claim:
 1. A method for imaging, comprising: a first step of using animager to perform a first set of measurements for generating a firstimage value, the first set of measurements including at least onemeasurement of an intensity of a first radiation ray bundle from a firstscene region, the first radiation ray bundle having a first chief ray ina reference frame of the imager, the imager having a first intensitysensitivity characteristic with respect to radiation ray bundles havingthe first chief ray, and the imager having a first dynamic range withrespect to intensities of the radiation ray bundles having the firstchief ray; a second step of using the imager to perform a set of secondmeasurements for generating a second image value, the second set ofmeasurements comprising at least one measurement of an intensity of asecond radiation ray bundle emanating from the first scene region, thesecond radiation ray bundle having a second chief ray in the referenceframe of the imager, the second chief ray being different from the firstchief ray, the imager having a second intensity sensitivitycharacteristic with respect to radiation ray bundles having the secondchief ray, the second intensity sensitivity characteristic beingdifferent from the first intensity sensitivity characteristic, and theimager having a second dynamic range with respect to intensities of theradiation ray bundles having the second chief ray; and applying amosaicing operation to the first and second image values, for generatinga third image value having associated therewith a third dynamic range ofthe imager with respect to at least one of the intensities of the firstand second radiation ray bundles, the third dynamic range being greaterthan at least one of the first and second dynamic ranges of the imager.2. A method according to claim 1, wherein the first radiation ray bundlecomprises a first electromagnetic radiation ray bundle, the secondradiation ray bundle comprising a second electromagnetic radiation raybundle.
 3. A method according to claim 1, further comprising one of:rotating the imager with respect to the first scene region between thefirst and second steps; and translating the imager with respect to thefirst scene region between the first and second steps.
 4. A methodaccording to claim 1, further comprising calibrating the imager, thecalibrating step comprising: using the imager to perform measurements ofintensities of a first plurality of radiation ray bundles having thefirst chief ray, for generating a first set of calibration measurementvalues; determining a first estimate of the first intensity sensitivitycharacteristic by determining one of: a) a sum of the first set ofcalibration measurement values, and b) a mean of the first set ofcalibration measurement values; using the imager to perform measurementsof intensities of a second plurality of radiation ray bundles having thesecond chief ray, for generating a second set of calibration measurementvalues; and determining a second estimate of the second intensitysensitivity characteristic by determining one of: a) a sum of the secondset of calibration measurement values, and D) a mean of the second setof calibration measurement values.
 5. A method according to claim 1,further comprising calibrating the imager the calibrating stepcomprising: using the imager to perform a third set of measurements forgenerating a fourth image value, the third set of measurementscomprising at least one measurement of an intensity of a third radiationray bundle, the third radiation ray bundle emanating from a second sceneregion and having the first chief ray; using the imager to perform afourth set of measurements for generating a fifth image value, thefourth set of measurements comprising at least one measurement of anintensity of a fourth radiation ray bundle, the fourth radiation raybundle emanating from the second scene region and having the secondchief ray; and estimating a relationship between the first and secondintensity sensitivity characteristics, the estimating step comprisingdetermining one of: a) a difference of the fourth and fifth imagevalues, and b) a ratio of the fourth and fifth image values.
 6. A methodaccording to claim 1, further comprising: using the imager to perform athird set of measurements for generating a fourth image value, the thirdset of measurements comprising at least one measurement of an intensityof at least one selected spectral component of a third radiation raybundle, the third radiation ray bundle emanating from the first sceneregion and having a third chief ray in the reference frame of theimager, the imager having a first spectral sensitivity characteristicwith respect to radiation ray bundles having the third chief ray, thefirst spectral sensitivity characteristic comprising a bandpasscharacteristic having a first wavelength sensitivity band, and the atleast one selected spectral component of the third radiation ray bundlehaving a wavelength within the first wavelength sensitivity band; andusing the imager to perform a fourth set of measurements for generatinga fifth image value, the fourth set of measurements comprising at leastone measurement of an intensity of at least one selected spectralcomponent of a fourth radiation ray bundle, the fourth radiation raybundle emanating from the first scene region and having a fourth chiefray in the reference frame of the imager, the fourth chief ray beingdifferent from the third chief ray, the imager having a second spectralsensitivity characteristic with respect to radiation ray bundles havingthe fourth chief ray, the second spectral sensitivity characteristiccomprising a bandpass characteristic having a second wavelengthsensitivity band, the at least one selected spectral component of thefourth radiation ray bundle having a wavelength within the secondwavelength sensitivity band, and the second wavelength sensitivity bandbeing different from the first wavelength sensitivity band.
 7. A methodaccording to claim 1, further comprising: using the imager to perform athird set of measurements for generating a fourth iamge value, the thirdset of measurements comprising at least one measurement of an intensityof at least one selected polarization component of a third radiation raybundle emanating from the first scene region, the third radiation raybundle having a third chief ray in the reference frame of the imager,the imager having a first polarization sensitivity characteristic withrespect to radiation ray bundles having the third chief ray, the firstpolarization sensitivity characteristic comprising reduced sensitivityto radiation ray components having polarization angles outside a firstangular range, and the at least one selected polarization component ofthe third radiation ray bundle having a polarization angle within thefirst angular range; and using the imager to perform a fourth set ofmeasurements for generating a fifth image value, the fourth set ofmeasurements comprising at least one measurement of an intensity of atleast one selected polarization component of a fourth radiation raybundle, the fourth radiation ray bundle emanating from the first sceneregion and having a fourth chief ray in the reference frame of theimager, the fourth chief ray being different from the third chief ray,the imager having a second polarization sensitivity characteristic withrespect to radiation ray bundles having the fourth chief ray, the secondpolarization sensitivity characteristic comprising reduced sensitivityto signal components having polarization angles outside a second angularrange, the at least one selected polarization component of the fourthradiation ray bundle having a polarization angle within the secondangular range, and the second angular range being different from thefirst angular range.
 8. A method according to claim 1, furthercomprising: using the imager to perform a third set of measurements forgenerating a fourth image value, the third set of measurementscomprising at least one measurement of an intensity of a third radiationray bundle emanating from the first scene region, the third radiationray bundle having a third chief ray in the reference frame of theimager, and the imager having a first focal characteristic with respectto radiation ray bundles having the third chief ray, the first focalcharacteristic comprising a first focal distance; and using the imagerto perform a fourth set of measurements for generating a fifth imagevalue, the fourth set of measurements comprising at least onemeasurement of an intensity of a fourth radiation ray bundle emanatingfrom the first scene region, the fourth radiation ray bundle having afourth chief ray in the reference frame of the imager, the fourth chiefray being different from the third chief ray, the imager having a secondfocal characteristic with respect to radiation ray bundles having thefourth chief ray, the second focal characteristic comprising a secondfocal distance, and the second focal distance being different from thefirst focal distance.
 9. A method for imaging, comprising: a first stepof using an imager to perform a first set of measurements for generatinga first image value, the first set of measurements comprising at leastone measurement of an intensity of at least one selected polarizationcomponent of a first radiation ray bundle emanating from a first sceneregion, the first radiation ray bundle having a first chief ray in areference flame of the imager, the imager having a first polarizationsensitivity characteristic with respect to radiation ray bundles havingthe first chief ray, and the first polarization sensitivitycharacteristic comprising reduced sensitivity to signal componentshaving polarization angles outside a first angular range, the at leastone selected polarization component of the first radiation ray bundlehaving a polarization angle within the first angular range; a secondstep of using the imager to perform a second set of measurements forgenerating a second image value, the second set of measurementscomprising at least one measurement of an intensity of at least oneselected polarization component of a second radiation ray bundleemanating from the first scene region, the second radiation ray bundlehaving a second chief ray in the reference frame of the imager, thesecond chief ray being different from the first chief ray, the imagerhaving a second polarization sensitivity characteristic with respect toradiation ray bundles having the second chief ray, the secondpolarization sensitivity characteristic comprising reduced sensitivityto signal components having polarization angles outside a second angularrange, the at least one selected polarization component of the secondradiation ray bundle having a polarization angle within the secondangular range, and the second angular range being different from thefirst angular range; a third step of moving the imager, comprising oneof: rotating the imager with respect to the first scene region betweenthe first and second steps; and translating the imager with respect tothe first scene region between the first and second steps; and using thefirst and second image values to determine a polarization state of oneof the first and second radiation ray bundles.
 10. A method according toclaim 9, the first radiation ray bundle comprising a firstelectromagnetic radiation ray bundle, and the second radiation raybundle comprising a second electromagnetic radiation ray bundle.
 11. Anapparatus for imaging, comprising: a first imager for performing a firstset of measurements for generating a first image value, the first set ofmeasurements comprising at least one measurement of an intensity of afirst radiation ray bundle emanating from a first scene region, thefirst radiation ray bundle having a first chief ray in a reference frameof the imager, the imager having a first intensity sensitivitycharacteristic with respect to radiation ray bundles having the firstchief ray, the imager having a first dynamic range with respect tointensities of the radiation ray bundles having the first chief ray; asecond imager for performing a second set of measurements for generatinga second image value, the second set of measurements comprising at leastone measurement of an intensity of a second radiation ray bundleemanating from the first scene region, the second radiation ray bundlehaving a second chief ray in the reference frame of the imager, thesecond chief ray being different from the first chief ray, the imagerhaving a second intensity sensitivity characteristic with respect toradiation ray bundles having the second chief ray, the second intensitysensitivity characteristic being different from the first intensitysensitivity characteristic, the imager having a second dynamic rangewith respect to intensities of the radiation signal sets having thesecond chief ray; and a processor for applying a mosaicing operation tothe first and second measurement values, for generating a third imagevalue having associated therewith a third dynamic range with respect toat least one of the intensities of the first and second radiation raybundles, the third dynamic range being greater than at least one of thefirst and second dynamic ranges.
 12. An apparatus according to claim 11,wherein the first radiation ray bundle comprises a first electromagneticradiation ray bundle, the second radiation ray bundle comprising asecond electromagnetic radiation ray bundle.
 13. An apparatus accordingto claim 11, wherein the second imager is the first imager, the firstset of measurements being performed no later than a first time, thesecond set of measurements being performed no earlier than a secondtime, the second time being later than the first time, and the apparatusfurther comprising one of: an arrangement for rotating the first imagerwith respect to the first and second scene regions between the first andsecond times; and an arrangement for translating the first imager withrespect to the first and second scene regions between the first andsecond times.
 14. An apparatus according to claim 11, furthercomprising: a processor for using the imager to measure intensities of afirst plurality of radiation ray bundles having the first chief ray, forgenerating a first set of calibration measurement values; a processorfor determining a first estimate of the first intensity sensitivitycharacteristic, comprising one of: a) a processor for determining a sumof the first set of calibration measurement values, and b) a processorfor determining a mean of the first set of calibration measurementvalues; a processor for using the imager to measure intensities of asecond plurality of radiation ray bundles having the second chief ray,for generating a second set of calibration measurement values; and aprocessor for determining a second estimate of the second intensitysensitivity characteristic, comprising one of: a) a processor fordetermining a sum of the second set of calibration measurement values,and b) a processor for determining a mean of the second set ofcalibration measurement values.
 15. An apparatus according to claim 11,further comprising: a third imager for performing a third set ofmeasurements for generating a fourth image value, the third set ofmeasurements comprising at least one measurement of an intensity of athird radiation ray bundle emanating from a second scene region, and thethird radiation ray bundle having the first chief ray; a fourth imagerfor performing a fourth set of measurements for generating a fifth imagevalue, the fourth set of measurements comprising at least onemeasurement of an intensity of a fourth radiation ray bundle emanatingfrom the second scene region, the fourth radiation ray bundle having thesecond chief ray; and a processor for estimating a relationship betweenthe first and second intensity sensitivity characteristics, comprisingone of: a) a processor for determining a difference of the fourth andfifth image values, and b) a processor for determining a ratio of thefourth and fifth image values.
 16. An apparatus according to claim 11,further comprising: a third imager for performing a set of thirdmeasurements for generating a fourth image value, the third measurementset comprising at least one measurement of an intensity of at least oneselected spectral component of a third radiation ray bundle emanatingfrom the first scene region, the third radiation ray bundle having athird chief ray in the reference frame of the imager, the imager havinga first spectral sensitivity characteristic with respect to radiationray bundles having the third chief ray, the first spectral sensitivitycharacteristic comprising a bandpass characteristic having a firstwavelength sensitivity band, the at least one selected spectralcomponent of the third radiation ray bundle having a wavelength withinthe first wavelength sensitivity band; a fourth imager for performing afourth set of measurements for generating a fifth image value, thefourth set of measurements comprising at least one measurement of anintensity of at least one selected spectral component of a fourthradiation ray bundle emanating from the first scene region, the fourthradiation ray bundle having a fourth chief ray in the reference frame ofthe imager, the fourth chief ray being different from the third chiefray, the imager having a second spectral sensitivity characteristic withrespect to radiation ray bundles having the fourth chief ray, the secondspectral sensitivity characteristic comprising a bandpass characteristichaving a second wavelength sensitivity band, the at least one selectedspectral component of the fourth radiation ray bundle having awavelength within the second wavelength sensitivity band, and the secondwavelength sensitivity band being different from the first wavelengthsensitivity band.
 17. An apparatus according to claim 11, furthercomprising: a third imager for performing a third set of measurementsfor generating a fourth image value, the third set of measurementscomprising at least one measurement of an intensity of at least oneselected polarization component of a third radiation ray bundleemanating from the first scene region, the third radiation ray bundlehaving a third chief ray in the reference frame of the imager, theimager having a first polarization sensitivity characteristic withrespect to radiation ray bundles having the third chief ray, the firstpolarization sensitivity characteristic comprising reduced sensitivityto signal components having polarization angles outside a first angularrange, and the at least one selected polarization component of the thirdradiation ray bundle having a polarization angle within the firstangular range; a fourth imager for performing a fourth set ofmeasurements for generating a fifth image value, the fourth set ofmeasurements comprising at least one measurement of an intensity of atleast one selected polarization component of a fourth radiation raybundle emanating from the first scene region, the fourth radiation raybundle having a fourth chief ray in the reference frame of the imager,the fourth chief ray being different from the third chief ray, theimager having a second polarization sensitivity characteristic withrespect to radiation ray bundles having the fourth chief ray, the secondpolarization sensitivity characteristic comprising reduced sensitivityto signal components having polarization angles outside a second angularrange, the at least one selected polarization component of the fourthradiation ray bundle having a polarization angle within the secondangular range, and the second angular range being different from thefirst angular range.
 18. An apparatus according to claim 11, furthercomprising: a third imager for performing a third set of measurementsfor generating a fourth image value, the third set of measurementscomprising at least one measurement of an intensity of a third radiationray bundle emanating from the first scene region, the third radiationray bundle having a third chief ray in the reference frame of theimager, and the imager having a first focal characteristic with respectto radiation ray bundles having the third chief ray, the first focalcharacteristic comprising a first focal distance; a fourth imager forperforming a fourth set of measurements for generating a fifth imagevalue, the fourth set of measurements comprising at least onemeasurement of an intensity of a fourth radiation ray bundle emanatingfrom the first scene region, the fourth radiation ray bundle having afourth chief ray in the reference frame of the imager, the fourth chiefray being different from the third chief ray, the imager having a secondfocal characteristic with respect to radiation ray bundles having thefourth chief ray, the second focal characteristic comprising a secondfocal distance, and the second focal distance being different from thefirst focal distance.
 19. An apparatus for imaging, comprising: a firstimager for performing a first set of measurements for generating a firstimage value, the first set of measurements comprising at least onemeasurement of an intensity of at least one selected polarizationcomponent of a first radiation ray bundle emanating from a first sceneregion, the first radiation ray bundle having a first chief ray in areference frame of the imager, the imager having a first polarizationsensitivity characteristic with respect to radiation ray bundles havingthe first chief ray, the first polarization sensitivity characteristiccomprising reduced sensitivity to signal components having polarizationangles outside a first angular range, the at least one selectedpolarization component of the first radiation ray bundle having apolarization angle within the first angular range; a second imager forperforming a second set ofI measurements for generating a second imagevalue, the second set of measurements comprising at least onemeasurement of an intensity of at least one selected polarizationcomponent of a second radiation ray bundle emanating from the firstscene region, the second radiation ray bundle having a second chief rayin the reference frame of the imager, the second chief ray beingdifferent from the first chief ray, the imager having a secondpolarization sensitivity characteristic with respect to radiation raybundles having the second chief ray, the second polarization sensitivitycharacteristic comprising reduced sensitivity to signal componentshaving polarization angles outside a second angular range, the at leastone selected polarization component of the second radiation ray bundlehaving a polarization angle within the second angular range, and thesecond angular range being different from the first angular range,wherein the first set of measurements is performed no later than a firsttime, the second measurement set being performed no earlier than asecond time, the second time being later than the first time; a movementdevice comprising one of: an arrangement for rotating the first imagerwith respect to the first and second scene regions between the first andsecond times, and an arrangement for translating the first imager withrespect to the first and second scene regions between the first andsecond times; and a processor for using the first and second imagevalues to determine a polarization state of one of the first and secondradiation ray bundles.
 20. An apparatus according to claim 19, the firstradiation ray bundle comprising a first electromagnetic radiation raybundle, and the second radiation ray bundle comprising a secondelectromagnetic radiation bundle.